2019-12-09T20:13:24.601 INFO:root:teuthology version: 1.0.0-8b630f9 2019-12-09T20:13:24.601 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.DI6jp6.tmp -- --verbose --lock --description rgw/verify/{clusters/fixed-2.yaml frontend/civetweb.yaml msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml overrides.yaml proto/http.yaml rgw_pool_type/ec-profile.yaml tasks/{0-install.yaml cls.yaml ragweed.yaml s3tests-java.yaml s3tests.yaml swift.yaml} validater/valgrind.yaml} --name abhi-2019-12-09_15:07:12-rgw-wip-abhi-testing-2019-12-04-1433-distro-basic-smithi --block --owner scheduled_abhi@teuthology --archive /home/teuthworker/archive/abhi-2019-12-09_15:07:12-rgw-wip-abhi-testing-2019-12-04-1433-distro-basic-smithi/4585197 2019-12-09T20:13:24.645 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-09T20:13:24.720 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/abhi-2019-12-09_15:07:12-rgw-wip-abhi-testing-2019-12-04-1433-distro-basic-smithi/4585197 branch: wip-abhi-testing-2019-12-04-1433 description: rgw/verify/{clusters/fixed-2.yaml frontend/civetweb.yaml msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml overrides.yaml proto/http.yaml rgw_pool_type/ec-profile.yaml tasks/{0-install.yaml cls.yaml ragweed.yaml s3tests-java.yaml s3tests.yaml swift.yaml} validater/valgrind.yaml} email: null first_in_suite: false job_id: '4585197' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: abhi-2019-12-09_15:07:12-rgw-wip-abhi-testing-2019-12-04-1433-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos overrides: admin_socket: branch: wip-abhi-testing-2019-12-04-1433 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw lc debug interval: 10 global: mon client directed command retry: 5 ms inject socket failures: 5000 osd heartbeat grace: 40 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd shutdown pgref assert: true fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 827e03c8a6b1e17d93dcaf79724437bac291c35b valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: notcmalloc sha1: 827e03c8a6b1e17d93dcaf79724437bac291c35b rgw: compression type: random ec-data-pool: true erasure_code_profile: crush-failure-domain: osd k: 3 m: 1 name: testprofile frontend: civetweb storage classes: LUKEWARM, FROZEN s3tests: slow_backend: true thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-abhi-testing-2019-12-04-1433 sha1: 827e03c8a6b1e17d93dcaf79724437bac291c35b owner: scheduled_abhi@teuthology priority: 99 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 sha1: 827e03c8a6b1e17d93dcaf79724437bac291c35b suite: rgw suite_branch: wip-abhi-testing-2019-12-04-1433 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-04-1433/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 827e03c8a6b1e17d93dcaf79724437bac291c35b tasks: - install: flavor: notcmalloc - ceph: null - openssl_keys: null - rgw: client.0: valgrind: - --tool=memcheck - --max-threads=1024 - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - rgw/test_rgw_gc_log.sh - ragweed: client.0: default-branch: ceph-master rgw_server: client.0 stages: prepare,check - s3tests-java: client.0: force-branch: master force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: force-branch: public-access-qa git_remote: git://github.com/theanalyst/ rgw_server: client.0 - swift: client.0: rgw_server: client.0 teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.13068 2019-12-09T20:13:24.720 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-04-1433/qa; will attempt to use it 2019-12-09T20:13:24.721 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-04-1433/qa/tasks 2019-12-09T20:13:24.721 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-12-09T20:13:24.730 INFO:teuthology.task.internal:Checking packages... 2019-12-09T20:13:24.762 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'notcmalloc' and ceph hash '827e03c8a6b1e17d93dcaf79724437bac291c35b' 2019-12-09T20:13:24.762 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-12-09T20:13:24.762 INFO:teuthology.packaging:ref: None 2019-12-09T20:13:24.762 INFO:teuthology.packaging:tag: None 2019-12-09T20:13:24.762 INFO:teuthology.packaging:branch: wip-abhi-testing-2019-12-04-1433 2019-12-09T20:13:24.762 INFO:teuthology.packaging:sha1: 827e03c8a6b1e17d93dcaf79724437bac291c35b 2019-12-09T20:13:24.763 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&ref=wip-abhi-testing-2019-12-04-1433 2019-12-09T20:13:24.937 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-7936.g827e03c 2019-12-09T20:13:24.937 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-12-09T20:13:24.957 INFO:teuthology.task.internal:no buildpackages task found 2019-12-09T20:13:24.958 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-12-09T20:13:24.980 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-12-09T20:13:24.980 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-09T20:13:25.036 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-09T20:13:35.055 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-09T20:13:45.076 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-09T20:13:55.093 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-12-09T20:14:05.109 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-09T20:14:15.127 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-09T20:14:25.145 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2019-12-09T20:14:35.166 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_abhi@teuthology', 'description': '/home/teuthworker/archive/abhi-2019-12-09_15:07:12-rgw-wip-abhi-testing-2019-12-04-1433-distro-basic-smithi/4585197', 'machine_type': 'smithi'} 2019-12-09T20:14:35.182 DEBUG:teuthology.lock.ops:locked ubuntu@smithi067.front.sepia.ceph.com, ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:14:35.269 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-09T20:14:35.501 INFO:teuthology.provision.fog.smithi067:Scheduling deploy of centos 7.6 2019-12-09T20:14:35.506 INFO:teuthology.provision.fog.smithi081:Scheduling deploy of centos 7.6 2019-12-09T20:14:35.854 INFO:teuthology.orchestra.console:Power off smithi067 2019-12-09T20:14:35.855 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-12-09T20:14:35.885 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:35.917 INFO:teuthology.orchestra.console:Power off smithi081 2019-12-09T20:14:35.918 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-12-09T20:14:35.944 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:39.933 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:39.970 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:44.050 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:44.088 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:48.169 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:48.204 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:48.380 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2019-12-09T20:14:48.415 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2019-12-09T20:14:48.481 INFO:teuthology.orchestra.console:Power on smithi067 2019-12-09T20:14:48.482 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-12-09T20:14:48.506 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:48.517 INFO:teuthology.orchestra.console:Power on smithi081 2019-12-09T20:14:48.517 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-12-09T20:14:48.540 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:52.532 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:52.568 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:52.746 INFO:teuthology.orchestra.console:Power on for smithi067 completed 2019-12-09T20:14:52.848 INFO:teuthology.provision.fog.smithi067:Waiting for deploy to finish 2019-12-09T20:14:56.682 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi081.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-09T20:14:56.892 INFO:teuthology.orchestra.console:Power on for smithi081 completed 2019-12-09T20:14:56.993 INFO:teuthology.provision.fog.smithi081:Waiting for deploy to finish 2019-12-09T20:17:43.172 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:17:52.269 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:17:54.211 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:17:54.313 ERROR:teuthology.orchestra.connection:Error connecting to smithi067.front.sepia.ceph.com Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 437, in connect passphrase, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 749, in _auth raise saved_exception AuthenticationException: Authentication failed. 2019-12-09T20:18:01.325 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:18:10.377 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:18:19.434 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:18:28.489 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:18:37.546 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:18:46.602 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:18:55.657 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:19:01.319 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:19:04.713 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:19:09.993 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:19:19.049 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:19:28.106 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:19:42.250 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:19:42.443 INFO:teuthology.orchestra.run.smithi081:> hostname 2019-12-09T20:19:42.859 INFO:teuthology.orchestra.run.smithi081.stdout:smithi068 2019-12-09T20:19:42.859 INFO:teuthology.orchestra.run.smithi081:> grep smithi068 /etc/hosts 2019-12-09T20:19:42.900 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.68 smithi068.front.sepia.ceph.com smithi068 2019-12-09T20:19:42.900 INFO:teuthology.orchestra.run.smithi081:> sudo hostname smithi081 2019-12-09T20:19:43.008 INFO:teuthology.orchestra.run.smithi081:> sudo sed -i -e 's/smithi068/smithi081/g' /etc/hosts 2019-12-09T20:19:43.064 INFO:teuthology.orchestra.run.smithi081:> sudo sed -i -e 's/172.21.15.68/172.21.15.81/g' /etc/hosts 2019-12-09T20:19:43.118 INFO:teuthology.orchestra.run.smithi081:> sudo sed -i -e 's/smithi068/smithi081/g' /etc/hostname 2019-12-09T20:19:43.170 INFO:teuthology.orchestra.run.smithi081:> sudo hostname smithi081 2019-12-09T20:19:43.224 INFO:teuthology.provision.fog.smithi081:Deploy complete! 2019-12-09T20:19:43.226 DEBUG:teuthology.parallel:result is None 2019-12-09T20:20:34.124 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:20:34.321 INFO:teuthology.orchestra.run.smithi067:> hostname 2019-12-09T20:20:34.913 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067 2019-12-09T20:20:34.913 INFO:teuthology.orchestra.run.smithi067:> grep smithi067 /etc/hosts 2019-12-09T20:20:34.998 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67 smithi067.front.sepia.ceph.com smithi067 2019-12-09T20:20:34.999 INFO:teuthology.orchestra.run.smithi067:> sudo hostname smithi067 2019-12-09T20:20:35.159 INFO:teuthology.orchestra.run.smithi067:> sudo sed -i -e 's/smithi067/smithi067/g' /etc/hosts 2019-12-09T20:20:35.257 INFO:teuthology.orchestra.run.smithi067:> sudo sed -i -e 's/172.21.15.67/172.21.15.67/g' /etc/hosts 2019-12-09T20:20:35.336 INFO:teuthology.orchestra.run.smithi067:> sudo sed -i -e 's/smithi067/smithi067/g' /etc/hostname 2019-12-09T20:20:35.436 INFO:teuthology.orchestra.run.smithi067:> sudo hostname smithi067 2019-12-09T20:20:35.523 INFO:teuthology.provision.fog.smithi067:Deploy complete! 2019-12-09T20:20:35.525 DEBUG:teuthology.parallel:result is None 2019-12-09T20:20:35.690 INFO:teuthology.lock.ops:Checking smithi067.front.sepia.ceph.com 2019-12-09T20:20:35.691 INFO:teuthology.lock.ops:New key found. Updating... 2019-12-09T20:20:35.706 INFO:teuthology.lock.ops:Checking smithi081.front.sepia.ceph.com 2019-12-09T20:20:35.706 INFO:teuthology.lock.ops:New key found. Updating... 2019-12-09T20:20:35.720 INFO:teuthology.lock.ops:Updating [smithi067.front.sepia.ceph.com]: set os type and version on server 2019-12-09T20:20:35.720 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:20:35.721 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:20:35.808 INFO:teuthology.orchestra.run.smithi067:> true 2019-12-09T20:20:36.215 INFO:teuthology.orchestra.run.smithi067:> uname -m 2019-12-09T20:20:36.286 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2019-12-09T20:20:36.287 INFO:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2019-12-09T20:20:36.355 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Linux" 2019-12-09T20:20:36.356 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="7 (Core)" 2019-12-09T20:20:36.356 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2019-12-09T20:20:36.356 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2019-12-09T20:20:36.356 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="7" 2019-12-09T20:20:36.356 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-09T20:20:36.357 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2019-12-09T20:20:36.357 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-09T20:20:36.357 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.centos.org/" 2019-12-09T20:20:36.357 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-09T20:20:36.357 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:36.357 INFO:teuthology.orchestra.run.smithi067.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-09T20:20:36.357 INFO:teuthology.orchestra.run.smithi067.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-09T20:20:36.357 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-09T20:20:36.358 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-09T20:20:36.358 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:36.359 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2019-12-09T20:20:36.374 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: set os type and version on server 2019-12-09T20:20:36.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:20:36.375 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:20:36.466 INFO:teuthology.orchestra.run.smithi081:> true 2019-12-09T20:20:36.861 INFO:teuthology.orchestra.run.smithi081:> uname -m 2019-12-09T20:20:36.947 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2019-12-09T20:20:36.947 INFO:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2019-12-09T20:20:37.018 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Linux" 2019-12-09T20:20:37.018 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="7 (Core)" 2019-12-09T20:20:37.018 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2019-12-09T20:20:37.018 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2019-12-09T20:20:37.018 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="7" 2019-12-09T20:20:37.019 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-09T20:20:37.019 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2019-12-09T20:20:37.019 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-09T20:20:37.019 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.centos.org/" 2019-12-09T20:20:37.019 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-09T20:20:37.019 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:20:37.019 INFO:teuthology.orchestra.run.smithi081.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-09T20:20:37.020 INFO:teuthology.orchestra.run.smithi081.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-09T20:20:37.020 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-09T20:20:37.020 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-09T20:20:37.020 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:20:37.021 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2019-12-09T20:20:37.040 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2019-12-09T20:20:37.071 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi067.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2H9fW2R5/trFss4jyqgkL9w8xuX6elX8nGrQ+RL8mge/gi7oAgEIdS8nZZR6mOIMHJtAkRkbGdZRDeNedeUF+/vGVyLWDV6gjYBNid7Sm3NnLWhay2O5vtqilGTd9xa2RDN3KDkv3br154iaTnZGYhdvqZbdb3v8oUhnh5mto4vAvS2irdzADKDX81W83PM4uPuOB8VrWIkYxx795HAV0TUb+e3jdpNx0wnfR4aw+0BzBOgYLUhv8bDX3BwZSRJu8155N9evjOUg+K/G4hkjqXb+FUN9mmi32d4kKbGzTmX9awvr4p/iKR71hkRbXRWGBBJlakLjAAXphx7dxpy3b smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyft3l0JDlpGYgd1tPqZYy5sl97AbJOYYuBJUczv3OJJ+5Mjm9vmrNS3su0hKng/jm+JsyqGwS7WkjupHLH8StHd9CVBY7N6m+SLs+MR5mVaaOA8gg/BtaCZKkAT9CVNd4+z6wV7SBqawNfJnun5zEFZlgmadUtirejdtvvDOroc7MvW8SBXLm+AP5m8Kq5tjPq/nxA+O47yvCSuteI1XynmIC++xxGjbt2K/HBm8KA9Xqq6IA8Fy732r8NI3C/syMeMlSGHu7jrZTC8arYcfcspKwOd1mC4bfqviL7bP30BTxRoWCfE+NvNdcFFvdXVoHc5mmirjS1XTZmh9Zalf3 2019-12-09T20:20:37.071 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-09T20:20:37.119 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-12-09T20:20:37.154 INFO:teuthology.task.internal:Saving configuration 2019-12-09T20:20:37.173 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-12-09T20:20:37.202 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-12-09T20:20:37.217 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-12-09 20:14:35.173571', u'locked_by': u'scheduled_abhi@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi067.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC9bZu7Wgky2WzPPO9Vjxo8DBUm9tYayfpug+sZOQqDmAXeb4bo67rOB0MrDomeOYPpCWCA8wtrBbKd6pEvyA6A=', u'description': u'/home/teuthworker/archive/abhi-2019-12-09_15:07:12-rgw-wip-abhi-testing-2019-12-04-1433-distro-basic-smithi/4585197'} 2019-12-09T20:20:37.230 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-12-09 20:14:35.173590', u'locked_by': u'scheduled_abhi@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi081.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOTCr7CTiUpZ/rlnQiyKv1YH3J3dl3YSnaI917LwO0BGL8m2rYOnJWlnUaUNnRBnTjxw/8EvvpeLowqu5ClaxBk=', u'description': u'/home/teuthworker/archive/abhi-2019-12-09_15:07:12-rgw-wip-abhi-testing-2019-12-04-1433-distro-basic-smithi/4585197'} 2019-12-09T20:20:37.230 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-12-09T20:20:37.242 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-12-09T20:20:37.242 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2019-12-09T20:20:37.242 INFO:teuthology.run_tasks:Running task console_log... 2019-12-09T20:20:37.332 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-09T20:20:37.333 INFO:teuthology.run_tasks:Running task internal.connect... 2019-12-09T20:20:37.343 INFO:teuthology.task.internal:Opening connections... 2019-12-09T20:20:37.343 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:20:37.344 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:20:37.440 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:20:37.440 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:20:37.537 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-12-09T20:20:37.545 INFO:teuthology.orchestra.run.smithi081:> uname -m 2019-12-09T20:20:37.732 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2019-12-09T20:20:37.732 INFO:teuthology.orchestra.run.smithi081:> cat /etc/os-release 2019-12-09T20:20:37.780 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="CentOS Linux" 2019-12-09T20:20:37.780 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="7 (Core)" 2019-12-09T20:20:37.780 INFO:teuthology.orchestra.run.smithi081.stdout:ID="centos" 2019-12-09T20:20:37.780 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="rhel fedora" 2019-12-09T20:20:37.780 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="7" 2019-12-09T20:20:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-09T20:20:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2019-12-09T20:20:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-09T20:20:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.centos.org/" 2019-12-09T20:20:37.781 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-09T20:20:37.781 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:20:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-09T20:20:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-09T20:20:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-09T20:20:37.782 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-09T20:20:37.783 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:20:37.783 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2019-12-09T20:20:37.797 INFO:teuthology.orchestra.run.smithi067:> uname -m 2019-12-09T20:20:37.906 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2019-12-09T20:20:37.907 INFO:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2019-12-09T20:20:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Linux" 2019-12-09T20:20:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="7 (Core)" 2019-12-09T20:20:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2019-12-09T20:20:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2019-12-09T20:20:37.972 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="7" 2019-12-09T20:20:37.972 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-09T20:20:37.972 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2019-12-09T20:20:37.973 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-09T20:20:37.973 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.centos.org/" 2019-12-09T20:20:37.973 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-09T20:20:37.974 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:37.974 INFO:teuthology.orchestra.run.smithi067.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-09T20:20:37.974 INFO:teuthology.orchestra.run.smithi067.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-09T20:20:37.974 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-09T20:20:37.974 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-09T20:20:37.974 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:37.975 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2019-12-09T20:20:37.986 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-12-09T20:20:38.028 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-12-09T20:20:38.041 INFO:teuthology.task.internal:Checking for old test directory... 2019-12-09T20:20:38.041 INFO:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2019-12-09T20:20:38.044 INFO:teuthology.orchestra.run.smithi081:> test '!' -e /home/ubuntu/cephtest 2019-12-09T20:20:38.141 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-12-09T20:20:38.152 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-12-09T20:20:38.152 INFO:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2019-12-09T20:20:38.156 INFO:teuthology.orchestra.run.smithi081:> test -z $(ls -A /var/lib/ceph) 2019-12-09T20:20:38.270 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-12-09T20:20:38.313 INFO:teuthology.run_tasks:Running task kernel... 2019-12-09T20:20:38.330 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.1': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}} 2019-12-09T20:20:38.330 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-12-09T20:20:38.330 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:20:38.331 INFO:teuthology.orchestra.run.smithi081:> sudo package-cleanup -y --oldkernels 2019-12-09T20:20:39.240 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:20:40.360 INFO:teuthology.orchestra.run.smithi081.stdout:http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:40.360 INFO:teuthology.orchestra.run.smithi081.stdout:Trying other mirror. 2019-12-09T20:20:40.361 INFO:teuthology.orchestra.run.smithi081.stdout:To address this issue please refer to the below wiki article 2019-12-09T20:20:40.361 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:20:40.361 INFO:teuthology.orchestra.run.smithi081.stdout:https://wiki.centos.org/yum-errors 2019-12-09T20:20:40.361 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:20:40.361 INFO:teuthology.orchestra.run.smithi081.stdout:If above article doesn't help to resolve this issue please use https://bugs.centos.org/. 2019-12-09T20:20:40.361 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:20:40.532 INFO:teuthology.orchestra.run.smithi081.stdout:http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:40.533 INFO:teuthology.orchestra.run.smithi081.stdout:Trying other mirror. 2019-12-09T20:20:40.624 INFO:teuthology.orchestra.run.smithi081.stdout:http://mirror.oss.ou.edu/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:40.625 INFO:teuthology.orchestra.run.smithi081.stdout:Trying other mirror. 2019-12-09T20:20:40.768 INFO:teuthology.orchestra.run.smithi081.stdout:http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:40.768 INFO:teuthology.orchestra.run.smithi081.stdout:Trying other mirror. 2019-12-09T20:20:40.921 INFO:teuthology.orchestra.run.smithi081.stdout:http://mirror.pnl.gov/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:40.921 INFO:teuthology.orchestra.run.smithi081.stdout:Trying other mirror. 2019-12-09T20:20:42.137 INFO:teuthology.orchestra.run.smithi081.stdout:No old kernels to remove 2019-12-09T20:20:42.157 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:20:42.157 INFO:teuthology.orchestra.run.smithi067:> sudo package-cleanup -y --oldkernels 2019-12-09T20:20:43.069 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:20:43.990 INFO:teuthology.orchestra.run.smithi067.stdout:http://ftp.linux.ncsu.edu/pub/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:43.990 INFO:teuthology.orchestra.run.smithi067.stdout:Trying other mirror. 2019-12-09T20:20:43.990 INFO:teuthology.orchestra.run.smithi067.stdout:To address this issue please refer to the below wiki article 2019-12-09T20:20:43.990 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:43.991 INFO:teuthology.orchestra.run.smithi067.stdout:https://wiki.centos.org/yum-errors 2019-12-09T20:20:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:43.991 INFO:teuthology.orchestra.run.smithi067.stdout:If above article doesn't help to resolve this issue please use https://bugs.centos.org/. 2019-12-09T20:20:43.991 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:44.140 INFO:teuthology.orchestra.run.smithi067.stdout:http://dl.fedoraproject.org/pub/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:44.141 INFO:teuthology.orchestra.run.smithi067.stdout:Trying other mirror. 2019-12-09T20:20:44.232 INFO:teuthology.orchestra.run.smithi067.stdout:http://mirror.oss.ou.edu/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:44.232 INFO:teuthology.orchestra.run.smithi067.stdout:Trying other mirror. 2019-12-09T20:20:44.375 INFO:teuthology.orchestra.run.smithi067.stdout:http://mirrors.cat.pdx.edu/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:44.375 INFO:teuthology.orchestra.run.smithi067.stdout:Trying other mirror. 2019-12-09T20:20:44.528 INFO:teuthology.orchestra.run.smithi067.stdout:http://mirror.pnl.gov/epel/7/x86_64/repodata/d0f1134f0f9a3fed89fb7177d07a42aa429e0e37c0d6b3e1c907d638a5cd1164-updateinfo.xml.bz2: [Errno 14] HTTP Error 404 - Not Found 2019-12-09T20:20:44.528 INFO:teuthology.orchestra.run.smithi067.stdout:Trying other mirror. 2019-12-09T20:20:45.651 INFO:teuthology.orchestra.run.smithi067.stdout:No old kernels to remove 2019-12-09T20:20:45.670 INFO:teuthology.orchestra.run.smithi067:> uname -r 2019-12-09T20:20:45.753 INFO:teuthology.orchestra.run.smithi067.stdout:3.10.0-1062.4.1.el7.x86_64 2019-12-09T20:20:45.754 INFO:teuthology.task.kernel:Running kernel on smithi067: 3.10.0-1062.4.1.el7.x86_64 2019-12-09T20:20:45.754 INFO:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2019-12-09T20:20:46.031 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:20:46.081 INFO:teuthology.orchestra.run.smithi067.stdout:Determining fastest mirrors 2019-12-09T20:20:46.234 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:20:46.234 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: ftp.linux.ncsu.edu 2019-12-09T20:20:46.235 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:20:46.236 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.vcu.edu 2019-12-09T20:20:47.658 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:20:47.658 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:20:47.659 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package kernel.x86_64 0:3.10.0-1062.9.1.el7 will be installed 2019-12-09T20:20:48.819 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:20:49.601 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:49.601 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:20:49.606 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:49.606 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:20:49.606 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2019-12-09T20:20:49.606 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:20:49.607 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:20:49.607 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 3.10.0-1062.9.1.el7 updates 50 M 2019-12-09T20:20:49.607 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:49.607 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:20:49.607 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:20:49.607 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2019-12-09T20:20:49.607 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:20:49.608 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 50 M 2019-12-09T20:20:49.608 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 64 M 2019-12-09T20:20:49.608 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:20:49.613 INFO:teuthology.orchestra.run.smithi067.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-09T20:20:52.606 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:20:52.855 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:20:53.734 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:20:53.735 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:21:31.099 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-3.10.0-1062.9.1.el7.x86_64 1/1 2019-12-09T20:21:31.808 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-3.10.0-1062.9.1.el7.x86_64 1/1 2019-12-09T20:21:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:21:31.809 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:21:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: kernel.x86_64 0:3.10.0-1062.9.1.el7 2019-12-09T20:21:31.809 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:21:31.809 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:21:31.855 INFO:teuthology.orchestra.run.smithi067:> rpm -q kernel --last | head -n 1 2019-12-09T20:21:32.210 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-3.10.0-1062.9.1.el7.x86_64 Mon 09 Dec 2019 08:20:56 PM UTC 2019-12-09T20:21:32.212 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:21:32.212 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.4.1.el7.x86_64 Expected: 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:21:32.212 INFO:teuthology.orchestra.run.smithi081:> uname -r 2019-12-09T20:21:32.307 INFO:teuthology.orchestra.run.smithi081.stdout:3.10.0-1062.4.1.el7.x86_64 2019-12-09T20:21:32.308 INFO:teuthology.task.kernel:Running kernel on smithi081: 3.10.0-1062.4.1.el7.x86_64 2019-12-09T20:21:32.308 INFO:teuthology.orchestra.run.smithi081:> sudo yum install -y kernel 2019-12-09T20:21:32.551 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:21:32.593 INFO:teuthology.orchestra.run.smithi081.stdout:Determining fastest mirrors 2019-12-09T20:21:32.751 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:21:32.752 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: ftp.linux.ncsu.edu 2019-12-09T20:21:32.753 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:21:32.753 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.vcu.edu 2019-12-09T20:21:34.451 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:21:34.452 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:21:34.452 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package kernel.x86_64 0:3.10.0-1062.9.1.el7 will be installed 2019-12-09T20:21:35.715 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:21:36.607 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:21:36.607 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:21:36.608 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:21:36.609 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:21:36.609 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2019-12-09T20:21:36.609 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:21:36.609 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:21:36.609 INFO:teuthology.orchestra.run.smithi081.stdout: kernel x86_64 3.10.0-1062.9.1.el7 updates 50 M 2019-12-09T20:21:36.609 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:21:36.610 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:21:36.610 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:21:36.610 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2019-12-09T20:21:36.610 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:21:36.611 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 50 M 2019-12-09T20:21:36.611 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 64 M 2019-12-09T20:21:36.611 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:21:36.613 INFO:teuthology.orchestra.run.smithi081.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-09T20:21:39.755 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:21:39.958 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:21:40.845 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:21:40.845 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:22:16.149 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : kernel-3.10.0-1062.9.1.el7.x86_64 1/1 2019-12-09T20:22:16.710 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : kernel-3.10.0-1062.9.1.el7.x86_64 1/1 2019-12-09T20:22:16.710 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:22:16.710 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:22:16.710 INFO:teuthology.orchestra.run.smithi081.stdout: kernel.x86_64 0:3.10.0-1062.9.1.el7 2019-12-09T20:22:16.710 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:22:16.710 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:22:16.760 INFO:teuthology.orchestra.run.smithi081:> rpm -q kernel --last | head -n 1 2019-12-09T20:22:17.165 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-3.10.0-1062.9.1.el7.x86_64 Mon 09 Dec 2019 08:21:43 PM UTC 2019-12-09T20:22:17.165 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:22:17.166 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.4.1.el7.x86_64 Expected: 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:22:17.166 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-12-09T20:22:17.166 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-12-09T20:22:17.166 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-12-09T20:22:17.166 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-12-09T20:22:17.166 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2019-12-09T20:22:17.167 INFO:teuthology.orchestra.run.smithi067:> rpm -q kernel --last | head -n 1 2019-12-09T20:22:17.538 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-3.10.0-1062.9.1.el7.x86_64 Mon 09 Dec 2019 08:20:56 PM UTC 2019-12-09T20:22:17.540 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:22:17.540 INFO:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2 2019-12-09T20:22:17.702 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2 2019-12-09T20:22:17.702 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2019-12-09T20:22:17.703 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.02 2019-12-09T20:22:17.703 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 0.76.el7.centos 2019-12-09T20:22:17.703 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2019-12-09T20:22:17.703 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Tue 05 Nov 2019 03:41:14 PM UTC 2019-12-09T20:22:17.704 INFO:teuthology.orchestra.run.smithi067.stdout:Group : System Environment/Base 2019-12-09T20:22:17.704 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 0 2019-12-09T20:22:17.704 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2019-12-09T20:22:17.704 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-12-09T20:22:17.704 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-12-09T20:22:17.705 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-12-09T20:22:17.705 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : kbuilder.bsys.centos.org 2019-12-09T20:22:17.705 INFO:teuthology.orchestra.run.smithi067.stdout:Relocations : (not relocatable) 2019-12-09T20:22:17.705 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : CentOS BuildSystem 2019-12-09T20:22:17.706 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2019-12-09T20:22:17.706 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2019-12-09T20:22:17.706 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-12-09T20:22:17.706 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2019-12-09T20:22:17.707 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:22:17.707 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-12-09T20:22:17.707 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2019-12-09T20:22:17.707 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2019-12-09T20:22:17.708 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2019-12-09T20:22:17.710 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-12-09T20:22:17.710 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:22:17.710 INFO:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-12-09T20:22:18.229 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2019-12-09T20:22:18.397 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:22:18.416 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.9.1.el7.x86_64.img 2019-12-09T20:22:18.751 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-12-09T20:22:18.791 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-12-09T20:22:18.833 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-12-09T20:22:18.859 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-12-09T20:22:18.896 INFO:teuthology.orchestra.run.smithi067.stderr:Found linux image: /boot/vmlinuz-0-rescue-a1739c9c60b14f0891126d2e603bb003 2019-12-09T20:22:18.909 INFO:teuthology.orchestra.run.smithi067.stderr:Found initrd image: /boot/initramfs-0-rescue-a1739c9c60b14f0891126d2e603bb003.img 2019-12-09T20:22:20.076 INFO:teuthology.orchestra.run.smithi067.stderr:done 2019-12-09T20:22:20.083 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:22:20.146 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.m6XHpgURB6 2019-12-09T20:22:20.146 INFO:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.m6XHpgURB6 2019-12-09T20:22:20.217 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.m6XHpgURB6 2019-12-09T20:22:20.336 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.m6XHpgURB6 is 6KB 2019-12-09T20:22:20.392 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.m6XHpgURB6 2019-12-09T20:22:20.443 INFO:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 0 2019-12-09T20:22:20.620 INFO:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2019-12-09T20:22:20.624 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2019-12-09T20:22:20.624 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi081.front.sepia.ceph.com, path=None, version=distro) 2019-12-09T20:22:20.624 INFO:teuthology.orchestra.run.smithi081:> rpm -q kernel --last | head -n 1 2019-12-09T20:22:20.983 INFO:teuthology.orchestra.run.smithi081.stdout:kernel-3.10.0-1062.9.1.el7.x86_64 Mon 09 Dec 2019 08:21:43 PM UTC 2019-12-09T20:22:20.985 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:22:20.987 INFO:teuthology.orchestra.run.smithi081:> sudo rpm -qi grub2 2019-12-09T20:22:21.104 INFO:teuthology.orchestra.run.smithi081.stdout:Name : grub2 2019-12-09T20:22:21.105 INFO:teuthology.orchestra.run.smithi081.stdout:Epoch : 1 2019-12-09T20:22:21.105 INFO:teuthology.orchestra.run.smithi081.stdout:Version : 2.02 2019-12-09T20:22:21.105 INFO:teuthology.orchestra.run.smithi081.stdout:Release : 0.76.el7.centos 2019-12-09T20:22:21.105 INFO:teuthology.orchestra.run.smithi081.stdout:Architecture: x86_64 2019-12-09T20:22:21.105 INFO:teuthology.orchestra.run.smithi081.stdout:Install Date: Tue 05 Nov 2019 03:41:14 PM UTC 2019-12-09T20:22:21.106 INFO:teuthology.orchestra.run.smithi081.stdout:Group : System Environment/Base 2019-12-09T20:22:21.106 INFO:teuthology.orchestra.run.smithi081.stdout:Size : 0 2019-12-09T20:22:21.106 INFO:teuthology.orchestra.run.smithi081.stdout:License : GPLv3+ 2019-12-09T20:22:21.106 INFO:teuthology.orchestra.run.smithi081.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-12-09T20:22:21.106 INFO:teuthology.orchestra.run.smithi081.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-12-09T20:22:21.107 INFO:teuthology.orchestra.run.smithi081.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-12-09T20:22:21.107 INFO:teuthology.orchestra.run.smithi081.stdout:Build Host : kbuilder.bsys.centos.org 2019-12-09T20:22:21.107 INFO:teuthology.orchestra.run.smithi081.stdout:Relocations : (not relocatable) 2019-12-09T20:22:21.107 INFO:teuthology.orchestra.run.smithi081.stdout:Packager : CentOS BuildSystem 2019-12-09T20:22:21.107 INFO:teuthology.orchestra.run.smithi081.stdout:Vendor : CentOS 2019-12-09T20:22:21.107 INFO:teuthology.orchestra.run.smithi081.stdout:URL : http://www.gnu.org/software/grub/ 2019-12-09T20:22:21.107 INFO:teuthology.orchestra.run.smithi081.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-12-09T20:22:21.108 INFO:teuthology.orchestra.run.smithi081.stdout:Description : 2019-12-09T20:22:21.108 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:22:21.108 INFO:teuthology.orchestra.run.smithi081.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-12-09T20:22:21.108 INFO:teuthology.orchestra.run.smithi081.stdout:customizable bootloader with modular architecture. It supports a rich 2019-12-09T20:22:21.108 INFO:teuthology.orchestra.run.smithi081.stdout:variety of kernel formats, file systems, computer architectures and 2019-12-09T20:22:21.108 INFO:teuthology.orchestra.run.smithi081.stdout:hardware devices. 2019-12-09T20:22:21.110 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-12-09T20:22:21.110 INFO:teuthology.task.kernel:Updating grub on smithi081 to boot 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:22:21.110 INFO:teuthology.orchestra.run.smithi081:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-12-09T20:22:21.573 INFO:teuthology.orchestra.run.smithi081.stderr:Generating grub configuration file ... 2019-12-09T20:22:21.745 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:22:21.761 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.9.1.el7.x86_64.img 2019-12-09T20:22:22.021 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-12-09T20:22:22.045 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-12-09T20:22:22.085 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-12-09T20:22:22.099 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-12-09T20:22:22.128 INFO:teuthology.orchestra.run.smithi081.stderr:Found linux image: /boot/vmlinuz-0-rescue-a1739c9c60b14f0891126d2e603bb003 2019-12-09T20:22:22.137 INFO:teuthology.orchestra.run.smithi081.stderr:Found initrd image: /boot/initramfs-0-rescue-a1739c9c60b14f0891126d2e603bb003.img 2019-12-09T20:22:23.272 INFO:teuthology.orchestra.run.smithi081.stderr:done 2019-12-09T20:22:23.275 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:22:23.383 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.NBuzee73Dv 2019-12-09T20:22:23.383 INFO:teuthology.orchestra.run.smithi081:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NBuzee73Dv 2019-12-09T20:22:23.457 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.NBuzee73Dv 2019-12-09T20:22:23.606 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.NBuzee73Dv is 6KB 2019-12-09T20:22:23.675 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.NBuzee73Dv 2019-12-09T20:22:23.726 INFO:teuthology.orchestra.run.smithi081:> sudo grub2-set-default 0 2019-12-09T20:22:23.951 INFO:teuthology.orchestra.run.smithi081:> sudo shutdown -r now 2019-12-09T20:22:23.956 INFO:teuthology.misc:Re-opening connections... 2019-12-09T20:22:23.956 INFO:teuthology.misc:trying to connect to ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:22:23.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:22:23.958 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:22:24.045 INFO:teuthology.orchestra.run.smithi081:> true 2019-12-09T20:22:24.368 DEBUG:teuthology.misc:waited 0.41188287735 2019-12-09T20:22:25.369 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:22:25.369 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:22:25.370 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:00.024 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:00.025 DEBUG:teuthology.misc:waited 36.0689888 2019-12-09T20:23:01.025 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:01.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:01.026 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:04.092 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:04.093 DEBUG:teuthology.misc:waited 40.1369309425 2019-12-09T20:23:05.096 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:05.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:05.097 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:07.160 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:07.161 DEBUG:teuthology.misc:waited 43.2048909664 2019-12-09T20:23:08.162 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:08.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:08.162 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:10.232 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:10.233 DEBUG:teuthology.misc:waited 46.2769238949 2019-12-09T20:23:11.233 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:11.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:11.234 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:13.305 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:13.305 DEBUG:teuthology.misc:waited 49.3492188454 2019-12-09T20:23:14.306 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:14.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:14.306 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:16.376 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:16.377 DEBUG:teuthology.misc:waited 52.4209377766 2019-12-09T20:23:17.378 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:17.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:17.378 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:19.448 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:19.449 DEBUG:teuthology.misc:waited 55.4931027889 2019-12-09T20:23:20.450 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:20.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:20.451 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:22.521 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:22.521 DEBUG:teuthology.misc:waited 58.5655369759 2019-12-09T20:23:23.531 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:23.532 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:23.532 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:25.592 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:25.593 DEBUG:teuthology.misc:waited 61.6369547844 2019-12-09T20:23:26.594 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:26.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:26.595 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:28.664 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:28.665 DEBUG:teuthology.misc:waited 64.7090339661 2019-12-09T20:23:29.666 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:29.666 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:29.667 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:31.736 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:31.737 DEBUG:teuthology.misc:waited 67.7809867859 2019-12-09T20:23:32.738 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:32.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:32.739 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:34.809 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:34.809 DEBUG:teuthology.misc:waited 70.8532137871 2019-12-09T20:23:35.810 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:35.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:35.811 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:37.880 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:37.881 DEBUG:teuthology.misc:waited 73.9250538349 2019-12-09T20:23:38.882 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:38.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:38.882 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:40.952 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:40.953 DEBUG:teuthology.misc:waited 76.9969058037 2019-12-09T20:23:41.954 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:41.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:41.954 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:44.024 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:44.025 DEBUG:teuthology.misc:waited 80.0689339638 2019-12-09T20:23:45.025 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:45.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:45.026 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:47.096 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:47.097 DEBUG:teuthology.misc:waited 83.1410069466 2019-12-09T20:23:48.099 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:48.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:48.100 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:50.172 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:50.173 DEBUG:teuthology.misc:waited 86.2170858383 2019-12-09T20:23:51.176 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:51.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:51.177 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:53.240 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:53.241 DEBUG:teuthology.misc:waited 89.2849428654 2019-12-09T20:23:54.242 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:54.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:54.242 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:23:56.312 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2019-12-09T20:23:56.313 DEBUG:teuthology.misc:waited 92.3569009304 2019-12-09T20:23:57.313 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:23:57.313 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:23:57.314 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:24:57.315 DEBUG:teuthology.orchestra.remote:timed out 2019-12-09T20:24:57.316 DEBUG:teuthology.misc:waited 153.360257864 2019-12-09T20:24:58.317 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:24:58.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:24:58.318 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:24:58.430 INFO:teuthology.orchestra.run.smithi067:> true 2019-12-09T20:24:58.873 DEBUG:teuthology.misc:waited 154.9168818 2019-12-09T20:24:59.874 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-12-09T20:24:59.874 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "3.10.0-1062.9.1.el7.x86_64"... 2019-12-09T20:24:59.874 INFO:teuthology.orchestra.run.smithi067:> uname -r 2019-12-09T20:24:59.979 INFO:teuthology.orchestra.run.smithi067.stdout:3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:24:59.980 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:24:59.980 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-12-09T20:24:59.980 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-12-09T20:24:59.980 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "3.10.0-1062.9.1.el7.x86_64"... 2019-12-09T20:24:59.980 INFO:teuthology.orchestra.run.smithi081:> uname -r 2019-12-09T20:24:59.987 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-12-09T20:24:59.987 ERROR:teuthology.task.kernel:Saw exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 695, in wait_for_reboot assert not need_to_install(ctx, client, need_install[client]), \ File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 184, in need_to_install stdout=uname_fp, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 428, in run r.execute() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 98, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 493, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 806, in open_session timeout=timeout, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 933, in open_channel raise e error: [Errno 104] Connection reset by peer 2019-12-09T20:25:00.989 INFO:teuthology.misc:Re-opening connections... 2019-12-09T20:25:00.989 INFO:teuthology.misc:trying to connect to ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:25:00.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:25:00.990 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:25:01.098 INFO:teuthology.orchestra.run.smithi081:> true 2019-12-09T20:25:01.598 DEBUG:teuthology.misc:waited 0.60847902298 2019-12-09T20:25:02.599 INFO:teuthology.misc:trying to connect to ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:25:02.600 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:25:02.601 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:25:02.697 INFO:teuthology.orchestra.run.smithi067:> true 2019-12-09T20:25:03.084 DEBUG:teuthology.misc:waited 2.09510207176 2019-12-09T20:25:04.085 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-12-09T20:25:04.085 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "3.10.0-1062.9.1.el7.x86_64"... 2019-12-09T20:25:04.085 INFO:teuthology.orchestra.run.smithi081:> uname -r 2019-12-09T20:25:04.191 INFO:teuthology.orchestra.run.smithi081.stdout:3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:25:04.191 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.9.1.el7.x86_64 2019-12-09T20:25:04.191 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-12-09T20:25:05.192 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-12-09T20:25:05.193 INFO:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-12-09T20:25:05.343 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2019-12-09T20:25:05.355 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2019-12-09T20:25:05.355 INFO:teuthology.orchestra.run.smithi081:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-12-09T20:25:05.505 INFO:teuthology.orchestra.run.smithi081.stdout:ttyS1 2019-12-09T20:25:05.522 INFO:teuthology.run_tasks:Running task internal.base... 2019-12-09T20:25:05.531 INFO:teuthology.task.internal:Creating test directory... 2019-12-09T20:25:05.531 INFO:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-12-09T20:25:05.535 INFO:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-12-09T20:25:05.624 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-12-09T20:25:05.650 INFO:teuthology.run_tasks:Running task internal.archive... 2019-12-09T20:25:05.663 INFO:teuthology.task.internal:Creating archive directory... 2019-12-09T20:25:05.663 INFO:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-12-09T20:25:05.667 INFO:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-12-09T20:25:05.761 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-12-09T20:25:05.775 INFO:teuthology.task.internal:Enabling coredump saving... 2019-12-09T20:25:05.776 INFO:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-09T20:25:05.779 INFO:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-09T20:25:05.880 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-09T20:25:05.895 INFO:teuthology.orchestra.run.smithi081.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-09T20:25:05.897 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-12-09T20:25:05.908 INFO:teuthology.task.internal:Configuring sudo... 2019-12-09T20:25:05.909 INFO:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-12-09T20:25:05.913 INFO:teuthology.orchestra.run.smithi081:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-12-09T20:25:06.030 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-12-09T20:25:06.051 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-12-09T20:25:06.051 INFO:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-12-09T20:25:06.057 INFO:teuthology.orchestra.run.smithi081:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-12-09T20:25:06.143 INFO:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-09T20:25:06.242 INFO:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-09T20:25:06.371 INFO:teuthology.orchestra.run.smithi081:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-09T20:25:06.464 INFO:teuthology.orchestra.run.smithi081:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-09T20:25:06.574 INFO:teuthology.orchestra.run.smithi081:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-12-09T20:25:06.681 INFO:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-09T20:25:06.795 INFO:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-09T20:25:06.919 INFO:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-09T20:25:07.014 INFO:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-09T20:25:07.132 INFO:teuthology.orchestra.run.smithi067:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-12-09T20:25:07.245 INFO:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2019-12-09T20:25:07.249 INFO:teuthology.orchestra.run.smithi081:> sudo service rsyslog restart 2019-12-09T20:25:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-12-09T20:25:07.414 INFO:teuthology.orchestra.run.smithi081.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-12-09T20:25:07.492 INFO:teuthology.run_tasks:Running task internal.timer... 2019-12-09T20:25:07.501 INFO:teuthology.task.internal:Starting timer... 2019-12-09T20:25:07.501 INFO:teuthology.run_tasks:Running task pcp... 2019-12-09T20:25:07.532 INFO:teuthology.run_tasks:Running task selinux... 2019-12-09T20:25:07.545 INFO:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2019-12-09T20:25:07.725 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: [ OK ] 2019-12-09T20:25:07.728 INFO:teuthology.orchestra.run.smithi081:> sudo service auditd rotate 2019-12-09T20:25:07.896 INFO:teuthology.orchestra.run.smithi081.stdout:Rotating logs: [ OK ] 2019-12-09T20:25:07.898 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-12-09T20:25:07.899 INFO:teuthology.orchestra.run.smithi081:> /usr/sbin/getenforce 2019-12-09T20:25:07.985 INFO:teuthology.orchestra.run.smithi081.stdout:Permissive 2019-12-09T20:25:07.985 INFO:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2019-12-09T20:25:08.081 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2019-12-09T20:25:08.081 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi081.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi067.front.sepia.ceph.com': 'permissive'} 2019-12-09T20:25:08.081 INFO:teuthology.orchestra.run.smithi081:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-12-09T20:25:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-09T20:25:08.201 INFO:teuthology.orchestra.run.smithi067:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-12-09T20:25:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-09T20:25:08.328 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-12-09T20:25:08.328 INFO:teuthology.orchestra.run.smithi081:> sudo /usr/sbin/setenforce permissive 2019-12-09T20:25:08.459 INFO:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2019-12-09T20:25:08.583 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-12-09T20:25:08.595 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2019-12-09T20:25:08.596 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2019-12-09T20:25:08.613 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-12-09T20:25:08.616 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi081.front.sepia.ceph.com,smithi067.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-12-09T20:25:09.021 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-12-09T20:25:10.290 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2019-12-09T20:25:13.265 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-12-09T20:25:13.293 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:13 +0000 (0:00:00.073) 0:00:00.073 ******* 2019-12-09T20:25:13.302 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:13 +0000 (0:00:00.013) 0:00:00.086 ******* 2019-12-09T20:25:16.625 INFO:teuthology.task.ansible.out: 2019-12-09T20:25:16.627 INFO:teuthology.task.ansible.out:TASK [Gathering Facts] ********************************************************* ok: [smithi067.front.sepia.ceph.com] 2019-12-09T20:25:16.717 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:16 +0000 (0:00:03.415) 0:00:03.501 ******* 2019-12-09T20:25:16.756 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] 2019-12-09T20:25:16.850 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:16 +0000 (0:00:00.128) 0:00:03.630 ******* 2019-12-09T20:25:17.204 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2019-12-09T20:25:17.206 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-09T20:25:17.247 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:17 +0000 (0:00:00.399) 0:00:04.030 ******* 2019-12-09T20:25:17.304 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:17 +0000 (0:00:00.043) 0:00:04.074 ******* 2019-12-09T20:25:17.993 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-12-09T20:25:17.995 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-12-09T20:25:18.040 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:18 +0000 (0:00:00.750) 0:00:04.824 ******* 2019-12-09T20:25:18.047 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-12-09T20:25:18.088 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:18 +0000 (0:00:00.047) 0:00:04.872 ******* 2019-12-09T20:25:18.094 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2019-12-09T20:25:18.095 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:18.137 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:18 +0000 (0:00:00.048) 0:00:04.921 ******* 2019-12-09T20:25:18.144 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:18.193 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:18 +0000 (0:00:00.055) 0:00:04.977 ******* 2019-12-09T20:25:18.613 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2019-12-09T20:25:18.615 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi081.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-09T20:25:18.654 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:18 +0000 (0:00:00.461) 0:00:05.438 ******* 2019-12-09T20:25:18.692 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:18 +0000 (0:00:00.038) 0:00:05.476 ******* 2019-12-09T20:25:20.186 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2019-12-09T20:25:20.187 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575923118.67-73940110966889/source", "state": "file", "uid": 0} 2019-12-09T20:25:20.230 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:20 +0000 (0:00:01.537) 0:00:07.014 ******* 2019-12-09T20:25:20.244 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575923118.71-6751353792274/source", "state": "file", "uid": 0} 2019-12-09T20:25:20.282 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:20 +0000 (0:00:00.050) 0:00:07.065 ******* 2019-12-09T20:25:20.833 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2019-12-09T20:25:20.834 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:20.868 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:21.179 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:21.180 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:21.511 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:21.523 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:21.856 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:21.868 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:22.233 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi081.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "change 2019-12-09T20:25:22.234 INFO:teuthology.task.ansible.out:d": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:22.583 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi067.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": 2019-12-09T20:25:22.585 INFO:teuthology.task.ansible.out:false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:22.893 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:22.934 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-09T20:25:23.066 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2019-12-09T20:25:23.076 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2019-12-09T20:25:23.313 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:23 +0000 (0:00:03.031) 0:00:10.097 ******* 2019-12-09T20:25:23.353 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:23 +0000 (0:00:00.039) 0:00:10.137 ******* 2019-12-09T20:25:24.312 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2019-12-09T20:25:24.314 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] 2019-12-09T20:25:24.331 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] 2019-12-09T20:25:24.451 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2019-12-09T20:25:24.456 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:24 +0000 (0:00:01.098) 0:00:11.235 ******* 2019-12-09T20:25:24.494 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:24 +0000 (0:00:00.042) 0:00:11.278 ******* 2019-12-09T20:25:24.965 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2019-12-09T20:25:24.966 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.216256", "end": "2019-12-09 20:25:24.947651", "rc": 0, "start": "2019-12-09 20:25:24.731395", "stderr": "", "stderr_lines": [], "stdout": "2c3c6d863a854d6a1e155fc475e8273d88b91491", "stdout_lines": ["2c3c6d863a854d6a1e155fc475e8273d88b91491"]} 2019-12-09T20:25:25.010 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.515) 0:00:11.794 ******* 2019-12-09T20:25:25.027 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.232879", "end": "2019-12-09 20:25:24.993707", "rc": 0, "start": "2019-12-09 20:25:24.760828", "stderr": "", "stderr_lines": [], "stdout": "2c3c6d863a854d6a1e155fc475e8273d88b91491", "stdout_lines": ["2c3c6d863a854d6a1e155fc475e8273d88b91491"]} 2019-12-09T20:25:25.069 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.059) 0:00:11.853 ******* 2019-12-09T20:25:25.346 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2019-12-09T20:25:25.346 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011915", "end": "2019-12-09 20:25:25.317325", "failed_when_result": false, "rc": 0, "start": "2019-12-09 20:25:25.305410", "stderr": "", "stderr_lines": [], "stdout": "7b088106a1031364e6869fc2888a0e2542579008", "stdout_lines": ["7b088106a1031364e6869fc2888a0e2542579008"]} 2019-12-09T20:25:25.384 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.314) 0:00:12.168 ******* 2019-12-09T20:25:25.411 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.413 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.017253", "end": "2019-12-09 20:25:25.383457", "failed_when_result": false, "rc": 0, "start": "2019-12-09 20:25:25.366204", "stderr": "", "stderr_lines": [], "stdout": "7b088106a1031364e6869fc2888a0e2542579008", "stdout_lines": ["7b088106a1031364e6869fc2888a0e2542579008"]} 2019-12-09T20:25:25.454 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.069) 0:00:12.238 ******* 2019-12-09T20:25:25.495 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.041) 0:00:12.279 ******* 2019-12-09T20:25:25.502 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.539 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.043) 0:00:12.323 ******* 2019-12-09T20:25:25.545 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2019-12-09T20:25:25.547 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.584 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.045) 0:00:12.368 ******* 2019-12-09T20:25:25.590 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2019-12-09T20:25:25.592 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.634 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.049) 0:00:12.418 ******* 2019-12-09T20:25:25.641 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2019-12-09T20:25:25.642 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.685 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.046) 0:00:12.465 ******* 2019-12-09T20:25:25.687 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-12-09T20:25:25.688 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.725 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.043) 0:00:12.509 ******* 2019-12-09T20:25:25.743 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.783 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.058) 0:00:12.567 ******* 2019-12-09T20:25:25.792 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.829 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.045) 0:00:12.613 ******* 2019-12-09T20:25:25.835 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2019-12-09T20:25:25.837 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.874 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.044) 0:00:12.658 ******* 2019-12-09T20:25:25.910 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2019-12-09T20:25:25.911 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:25:25.949 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:25 +0000 (0:00:00.074) 0:00:12.733 ******* 2019-12-09T20:25:26.166 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-12-09T20:25:26.166 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-09T20:25:26.225 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:26 +0000 (0:00:00.276) 0:00:13.009 ******* 2019-12-09T20:25:26.264 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-09T20:25:26.305 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:25:26 +0000 (0:00:00.079) 0:00:13.089 ******* 2019-12-09T20:25:26.638 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-12-09T20:25:26.698 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-12-09T20:25:26.958 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-12-09T20:25:27.021 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-12-09T20:25:27.285 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-12-09T20:25:27.358 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-12-09T20:25:27.631 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-12-09T20:25:27.695 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-12-09T20:25:27.990 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-12-09T20:25:28.044 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-12-09T20:25:28.337 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-12-09T20:25:28.345 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-12-09T20:25:28.651 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-12-09T20:25:28.668 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-12-09T20:25:29.005 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-12-09T20:25:29.016 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-12-09T20:25:29.359 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-12-09T20:25:29.378 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-12-09T20:25:29.695 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-12-09T20:25:29.738 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-12-09T20:25:30.013 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-12-09T20:25:30.067 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-12-09T20:25:30.370 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-12-09T20:25:30.446 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-12-09T20:25:30.727 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-12-09T20:25:30.805 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-12-09T20:25:31.087 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-12-09T20:25:31.151 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-12-09T20:25:31.438 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-12-09T20:25:31.491 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-12-09T20:25:31.807 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-12-09T20:25:31.840 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-12-09T20:25:32.155 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-12-09T20:25:32.192 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-12-09T20:25:32.500 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-12-09T20:25:32.541 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-12-09T20:25:32.865 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-12-09T20:25:32.912 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-12-09T20:25:33.233 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-12-09T20:25:33.249 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-12-09T20:25:33.581 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-12-09T20:25:33.616 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-12-09T20:25:33.914 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-12-09T20:25:33.968 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-12-09T20:25:34.272 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-12-09T20:25:34.327 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-12-09T20:25:34.622 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-12-09T20:25:34.677 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-12-09T20:25:34.972 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-12-09T20:25:35.030 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-12-09T20:25:35.330 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-12-09T20:25:35.343 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-12-09T20:25:35.685 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-12-09T20:25:35.707 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-12-09T20:25:36.022 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-12-09T20:25:36.055 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-12-09T20:25:36.364 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-12-09T20:25:36.407 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-12-09T20:25:36.704 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-12-09T20:25:36.770 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-12-09T20:25:37.047 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-12-09T20:25:37.123 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-12-09T20:25:37.410 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-12-09T20:25:37.465 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-12-09T20:25:37.755 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-12-09T20:25:37.832 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-12-09T20:25:38.105 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-12-09T20:25:38.203 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-12-09T20:25:38.455 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-12-09T20:25:38.565 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-12-09T20:25:38.798 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-12-09T20:25:38.894 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-12-09T20:25:39.135 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-12-09T20:25:39.241 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-12-09T20:25:39.453 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-09T20:25:39.592 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-09T20:25:39.806 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-12-09T20:25:39.968 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-12-09T20:25:40.151 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-09T20:25:40.322 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-09T20:25:40.501 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-12-09T20:25:40.669 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-12-09T20:25:40.854 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-12-09T20:25:41.027 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-12-09T20:25:41.193 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-12-09T20:25:41.401 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-12-09T20:25:41.521 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-12-09T20:25:41.768 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-12-09T20:25:41.864 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-09T20:25:42.089 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-09T20:25:42.219 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-12-09T20:25:42.441 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-12-09T20:25:42.526 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-12-09T20:25:42.785 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-12-09T20:25:42.886 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-09T20:25:43.139 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-09T20:25:43.218 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-12-09T20:25:43.473 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-12-09T20:25:43.578 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-12-09T20:25:43.841 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-12-09T20:25:43.926 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-12-09T20:25:44.183 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-12-09T20:25:44.289 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-12-09T20:25:44.525 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-12-09T20:25:44.613 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-12-09T20:25:44.875 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-12-09T20:25:44.974 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-12-09T20:25:45.241 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-12-09T20:25:45.305 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-12-09T20:25:45.558 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-12-09T20:25:45.657 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-12-09T20:25:45.890 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-12-09T20:25:46.009 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-12-09T20:25:46.218 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-12-09T20:25:46.330 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-12-09T20:25:46.572 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-12-09T20:25:46.650 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-12-09T20:25:46.922 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-12-09T20:25:47.006 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-12-09T20:25:47.244 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-12-09T20:25:47.343 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-12-09T20:25:47.583 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-12-09T20:25:47.659 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-12-09T20:25:47.916 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-12-09T20:25:48.012 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-12-09T20:25:48.276 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-12-09T20:25:48.360 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-12-09T20:25:48.620 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-12-09T20:25:48.702 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-09T20:25:48.945 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-09T20:25:49.056 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-12-09T20:25:49.303 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-12-09T20:25:49.364 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-12-09T20:25:49.641 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-12-09T20:25:49.717 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-12-09T20:25:49.996 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-12-09T20:25:50.072 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-12-09T20:25:50.352 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-12-09T20:25:50.431 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-12-09T20:25:50.709 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-12-09T20:25:50.755 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-12-09T20:25:51.064 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-12-09T20:25:51.086 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-12-09T20:25:51.415 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-12-09T20:25:51.428 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-12-09T20:25:51.751 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-12-09T20:25:51.763 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-12-09T20:25:52.069 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-12-09T20:25:52.112 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-12-09T20:25:52.405 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-12-09T20:25:52.429 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-12-09T20:25:52.711 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-12-09T20:25:52.765 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-12-09T20:25:53.030 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-12-09T20:25:53.106 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-12-09T20:25:53.342 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-12-09T20:25:53.470 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-12-09T20:25:53.704 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-12-09T20:25:53.816 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-12-09T20:25:54.037 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-12-09T20:25:54.168 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-12-09T20:25:54.369 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-12-09T20:25:54.539 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-12-09T20:25:54.733 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-12-09T20:25:54.734 INFO:teuthology.task.ansible.out: 2019-12-09T20:25:54.872 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-12-09T20:25:55.051 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-12-09T20:25:55.230 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-12-09T20:25:55.408 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-12-09T20:25:55.612 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-12-09T20:25:55.748 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-12-09T20:25:55.962 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-12-09T20:25:56.126 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-12-09T20:25:56.295 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-12-09T20:25:56.484 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-12-09T20:25:56.645 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-12-09T20:25:56.794 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-12-09T20:25:56.987 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-12-09T20:25:57.146 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-12-09T20:25:57.366 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-12-09T20:25:57.496 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-12-09T20:25:57.696 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-12-09T20:25:57.802 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-12-09T20:25:58.041 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-12-09T20:25:58.153 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-12-09T20:25:58.371 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-12-09T20:25:58.492 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-12-09T20:25:58.722 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-12-09T20:25:58.844 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-12-09T20:25:59.050 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-12-09T20:25:59.200 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-12-09T20:25:59.408 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-12-09T20:25:59.527 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-12-09T20:25:59.742 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-12-09T20:25:59.888 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-12-09T20:26:00.083 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-12-09T20:26:00.249 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-12-09T20:26:00.406 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-12-09T20:26:00.616 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-12-09T20:26:00.786 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-12-09T20:26:00.969 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-12-09T20:26:01.133 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-12-09T20:26:01.302 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-12-09T20:26:01.481 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-12-09T20:26:01.660 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-09T20:26:01.820 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-12-09T20:26:02.009 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10102, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 10102} 2019-12-09T20:26:02.176 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-09T20:26:02.540 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10102, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 10102} 2019-12-09T20:26:02.750 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "name": "rraja", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-12-09T20:26:02.877 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:02 +0000 (0:00:36.571) 0:00:49.661 ******* 2019-12-09T20:26:02.903 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-12-09T20:26:02.941 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:02 +0000 (0:00:00.063) 0:00:49.725 ******* 2019-12-09T20:26:03.000 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-12-09T20:26:03.005 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-12-09T20:26:03.007 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7 2019-12-09T20:26:03.008 INFO:teuthology.task.ansible.out:L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32 2019-12-09T20:26:03.010 INFO:teuthology.task.ansible.out:f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b 2019-12-09T20:26:03.011 INFO:teuthology.task.ansible.out:09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJy 2019-12-09T20:26:03.013 INFO:teuthology.task.ansible.out:w 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 2019-12-09T20:26:03.014 INFO:teuthology.task.ansible.out: 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf853 2019-12-09T20:26:03.016 INFO:teuthology.task.ansible.out:2237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36b 2019-12-09T20:26:03.017 INFO:teuthology.task.ansible.out:deb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, 2019-12-09T20:26:03.019 INFO:teuthology.task.ansible.out: {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsL 2019-12-09T20:26:03.020 INFO:teuthology.task.ansible.out:gffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}]}, "changed": false} 2019-12-09T20:26:03.062 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:03 +0000 (0:00:00.121) 0:00:49.846 ******* 2019-12-09T20:26:03.242 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-12-09T20:26:03.245 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "name": "rraja", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-12-09T20:26:03.377 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:03 +0000 (0:00:00.314) 0:00:50.161 ******* 2019-12-09T20:26:03.405 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-12-09T20:26:03.444 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:03 +0000 (0:00:00.067) 0:00:50.228 ******* 2019-12-09T20:26:03.495 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-12-09T20:26:03.499 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-12-09T20:26:03.500 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7 2019-12-09T20:26:03.502 INFO:teuthology.task.ansible.out:L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32 2019-12-09T20:26:03.503 INFO:teuthology.task.ansible.out:f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b 2019-12-09T20:26:03.505 INFO:teuthology.task.ansible.out:09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e 2019-12-09T20:26:03.510 INFO:teuthology.task.ansible.out:66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa 2019-12-09T20:26:03.511 INFO:teuthology.task.ansible.out:8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 2019-12-09T20:26:03.513 INFO:teuthology.task.ansible.out:6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f8 2019-12-09T20:26:03.514 INFO:teuthology.task.ansible.out:48846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8a 2019-12-09T20:26:03.516 INFO:teuthology.task.ansible.out:a01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalho 2019-12-09T20:26:03.517 INFO:teuthology.task.ansible.out:jr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}]}, "changed": false} 2019-12-09T20:26:03.556 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:03 +0000 (0:00:00.111) 0:00:50.340 ******* 2019-12-09T20:26:04.056 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-12-09T20:26:04.057 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com -> localhost] => {"after": "2c3c6d863a854d6a1e155fc475e8273d88b91491", "attempts": 1, "before": "2c3c6d863a854d6a1e155fc475e8273d88b91491", "changed": false, "remote_url_changed": false} 2019-12-09T20:26:04.093 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:04 +0000 (0:00:00.536) 0:00:50.877 ******* 2019-12-09T20:26:04.220 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com -> localhost] => {"after": "2c3c6d863a854d6a1e155fc475e8273d88b91491", "attempts": 1, "before": "2c3c6d863a854d6a1e155fc475e8273d88b91491", "changed": false, "remote_url_changed": false} 2019-12-09T20:26:04.253 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:04 +0000 (0:00:00.159) 0:00:51.037 ******* 2019-12-09T20:26:04.483 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-09T20:26:04.487 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-12-09T20:26:04.637 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-12-09T20:26:04.863 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-12-09T20:26:05.014 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-12-09T20:26:05.224 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-12-09T20:26:05.386 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-12-09T20:26:05.570 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-12-09T20:26:05.737 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-12-09T20:26:05.922 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-12-09T20:26:05.923 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-12-09T20:26:06.097 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-12-09T20:26:06.099 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-12-09T20:26:06.268 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-12-09T20:26:06.455 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-12-09T20:26:06.620 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-12-09T20:26:06.622 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-12-09T20:26:06.823 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-12-09T20:26:06.825 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-12-09T20:26:06.961 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-12-09T20:26:07.163 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-12-09T20:26:07.320 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-12-09T20:26:07.325 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:07.510 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-12-09T20:26:07.532 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:07.671 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-12-09T20:26:07.673 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:07.684 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:07.686 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:07.881 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-12-09T20:26:07.884 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-12-09T20:26:07.891 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:07.902 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:07.913 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:08.017 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-12-09T20:26:08.279 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-12-09T20:26:08.395 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-12-09T20:26:08.403 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-12-09T20:26:08.643 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-12-09T20:26:08.645 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-12-09T20:26:08.788 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-12-09T20:26:08.988 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-12-09T20:26:09.179 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-12-09T20:26:09.306 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-12-09T20:26:09.514 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-12-09T20:26:09.651 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-12-09T20:26:09.879 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-12-09T20:26:09.989 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-12-09T20:26:10.233 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-12-09T20:26:10.320 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-12-09T20:26:10.553 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-12-09T20:26:10.648 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-12-09T20:26:10.924 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-12-09T20:26:10.927 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-12-09T20:26:11.040 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-12-09T20:26:11.041 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-12-09T20:26:11.243 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-12-09T20:26:11.244 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-12-09T20:26:11.391 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-12-09T20:26:11.393 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-12-09T20:26:11.581 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-12-09T20:26:11.730 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-12-09T20:26:11.956 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-12-09T20:26:12.082 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-12-09T20:26:12.301 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-12-09T20:26:12.303 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-12-09T20:26:12.440 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-12-09T20:26:12.441 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-12-09T20:26:12.675 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-12-09T20:26:12.792 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-12-09T20:26:12.993 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-12-09T20:26:13.117 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-12-09T20:26:13.354 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-12-09T20:26:13.481 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-12-09T20:26:13.715 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-12-09T20:26:13.895 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-12-09T20:26:14.032 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-12-09T20:26:14.235 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-12-09T20:26:14.404 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-12-09T20:26:14.406 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-12-09T20:26:14.525 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-12-09T20:26:14.527 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-12-09T20:26:14.757 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-12-09T20:26:14.894 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-12-09T20:26:15.114 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-12-09T20:26:15.260 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-12-09T20:26:15.447 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-12-09T20:26:15.621 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-12-09T20:26:15.756 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-12-09T20:26:15.917 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-12-09T20:26:16.102 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-12-09T20:26:16.275 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-12-09T20:26:16.421 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-12-09T20:26:16.619 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-12-09T20:26:16.780 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-09T20:26:16.982 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-09T20:26:17.150 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-12-09T20:26:17.355 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-12-09T20:26:17.534 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-12-09T20:26:17.696 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-12-09T20:26:17.878 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-12-09T20:26:17.879 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-12-09T20:26:18.048 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-12-09T20:26:18.050 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-12-09T20:26:18.215 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-12-09T20:26:18.217 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-12-09T20:26:18.411 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-12-09T20:26:18.413 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-12-09T20:26:18.545 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-12-09T20:26:18.754 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-12-09T20:26:18.896 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-12-09T20:26:19.112 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-12-09T20:26:19.254 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-12-09T20:26:19.466 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-12-09T20:26:19.633 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2019-12-09T20:26:19.634 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-09T20:26:19.808 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2019-12-09T20:26:19.813 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-09T20:26:19.980 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-12-09T20:26:19.982 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-12-09T20:26:19.984 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-12-09T20:26:20.184 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-12-09T20:26:20.186 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-12-09T20:26:20.188 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-12-09T20:26:20.346 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-12-09T20:26:20.530 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-12-09T20:26:20.687 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-12-09T20:26:20.888 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-12-09T20:26:21.065 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-12-09T20:26:21.272 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-12-09T20:26:21.442 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-12-09T20:26:21.667 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-12-09T20:26:21.795 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-12-09T20:26:22.012 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-12-09T20:26:22.140 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-12-09T20:26:22.326 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-12-09T20:26:22.484 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-12-09T20:26:22.695 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-12-09T20:26:22.853 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-12-09T20:26:23.057 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-12-09T20:26:23.206 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-12-09T20:26:23.208 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-12-09T20:26:23.389 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-12-09T20:26:23.390 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-12-09T20:26:23.557 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-12-09T20:26:23.737 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-12-09T20:26:23.928 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-12-09T20:26:24.069 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-12-09T20:26:24.285 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-12-09T20:26:24.289 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-12-09T20:26:24.397 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-12-09T20:26:24.400 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-12-09T20:26:24.626 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-12-09T20:26:24.761 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-12-09T20:26:24.959 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-12-09T20:26:25.126 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-12-09T20:26:25.323 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-12-09T20:26:25.492 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-12-09T20:26:25.664 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-09T20:26:25.854 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-09T20:26:25.971 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6H 2019-12-09T20:26:25.973 INFO:teuthology.task.ansible.out:BvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-12-09T20:26:26.191 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-12-09T20:26:26.317 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-12-09T20:26:26.532 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-12-09T20:26:26.648 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-12-09T20:26:26.888 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-12-09T20:26:27.014 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-12-09T20:26:27.259 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-12-09T20:26:27.372 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-12-09T20:26:27.596 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-12-09T20:26:27.725 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-12-09T20:26:27.926 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-12-09T20:26:28.094 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-12-09T20:26:28.267 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-12-09T20:26:28.493 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-12-09T20:26:28.604 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-12-09T20:26:28.838 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-12-09T20:26:28.968 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-12-09T20:26:29.193 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-12-09T20:26:29.312 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-12-09T20:26:29.549 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-12-09T20:26:29.668 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-12-09T20:26:29.887 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-12-09T20:26:29.990 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-12-09T20:26:30.258 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-12-09T20:26:30.353 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-12-09T20:26:30.607 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-12-09T20:26:30.691 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-12-09T20:26:30.927 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-12-09T20:26:31.040 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-12-09T20:26:31.271 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-12-09T20:26:31.409 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-12-09T20:26:31.636 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-12-09T20:26:31.767 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-12-09T20:26:31.998 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-12-09T20:26:32.114 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-12-09T20:26:32.373 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-12-09T20:26:32.451 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-12-09T20:26:32.724 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-12-09T20:26:32.756 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-12-09T20:26:33.086 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-12-09T20:26:33.108 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-12-09T20:26:33.462 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-12-09T20:26:33.464 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-12-09T20:26:33.480 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-12-09T20:26:33.481 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-12-09T20:26:33.846 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-12-09T20:26:33.848 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-12-09T20:26:33.910 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-12-09T20:26:33.912 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-12-09T20:26:34.161 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-12-09T20:26:34.292 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-12-09T20:26:34.502 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-12-09T20:26:34.610 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-12-09T20:26:34.836 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-12-09T20:26:34.961 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-12-09T20:26:35.202 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-12-09T20:26:35.338 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-12-09T20:26:35.580 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-12-09T20:26:35.694 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-12-09T20:26:35.957 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-12-09T20:26:36.052 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-12-09T20:26:36.305 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-12-09T20:26:36.395 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-12-09T20:26:36.621 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-12-09T20:26:36.728 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-12-09T20:26:36.980 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-12-09T20:26:37.097 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-12-09T20:26:37.353 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-12-09T20:26:37.459 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-12-09T20:26:37.715 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-12-09T20:26:37.800 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-12-09T20:26:38.057 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-12-09T20:26:38.151 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-12-09T20:26:38.362 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-12-09T20:26:38.500 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-12-09T20:26:38.664 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-12-09T20:26:38.860 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-12-09T20:26:38.910 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2019-12-09T20:26:38.911 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-09T20:26:39.172 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2019-12-09T20:26:39.174 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-09T20:26:39.249 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2019-12-09T20:26:39.517 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2019-12-09T20:26:39.612 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2019-12-09T20:26:39.710 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:39 +0000 (0:00:35.455) 0:01:26.492 ******* 2019-12-09T20:26:39.757 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-09T20:26:39.759 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:39.772 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:39.786 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:39.787 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:39.799 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:39.817 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:39.824 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:39.836 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:39.881 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-09T20:26:39.883 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2019-12-09T20:26:39.991 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:39 +0000 (0:00:00.282) 0:01:26.775 ******* 2019-12-09T20:26:40.049 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-09T20:26:40.051 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:40.064 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:40.082 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:40.092 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:40.122 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:40.123 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:40.125 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:40.198 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-12-09T20:26:40.209 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:40.488 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-12-09T20:26:40.489 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:40.563 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-12-09T20:26:40.843 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-12-09T20:26:40.918 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-12-09T20:26:41.208 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-12-09T20:26:41.285 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-12-09T20:26:41.297 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.313 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.314 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.316 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.326 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.337 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.348 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.359 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.370 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.371 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.393 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.405 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.415 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.430 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.454 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.477 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.484 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.496 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.506 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.507 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.518 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.529 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.541 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.552 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.563 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.564 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-12-09T20:26:41.586 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.589 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.600 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.602 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.611 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.613 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.614 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.626 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.627 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.629 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.647 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.658 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.671 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.688 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.696 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin" 2019-12-09T20:26:41.698 INFO:teuthology.task.ansible.out:, "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.710 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.722 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.723 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.736 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.738 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.740 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.753 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.756 INFO:teuthology.task.ansible.out: skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.758 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.770 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.772 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.786 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.806 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.812 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.832 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.835 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.843 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.844 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.846 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.856 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.857 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.868 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.870 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.873 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.882 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.884 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.905 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.906 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.908 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.909 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.924 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.936 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.936 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.938 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.951 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.960 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.961 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.962 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:41.979 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.010 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.011 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.012 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.037 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.039 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.050 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.051 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.054 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.055 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.056 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.058 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.060 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.062 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.064 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.065 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.077 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.078 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.090 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.091 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.103 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.116 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.118 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.120 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.121 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.123 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.124 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.126 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.137 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.140 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.156 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.158 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.168 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.179 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.181 INFO:teuthology.task.ansible.out: 2019-12-09T20:26:42.215 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.216 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.218 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.219 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.247 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.248 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.249 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.252 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.297 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:42 +0000 (0:00:02.306) 0:01:29.082 ******* 2019-12-09T20:26:42.299 INFO:teuthology.task.ansible.out: 2019-12-09T20:26:42.306 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.308 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.311 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.311 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.313 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.326 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.327 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.372 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:42 +0000 (0:00:00.073) 0:01:29.156 ******* 2019-12-09T20:26:42.379 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-09T20:26:42.381 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.385 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.397 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.408 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.409 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.425 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.437 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.448 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.470 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.481 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.496 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.504 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.526 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.542 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.608 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:42 +0000 (0:00:00.236) 0:01:29.392 ******* 2019-12-09T20:26:42.646 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2019-12-09T20:26:42.648 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:26:42.682 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:42 +0000 (0:00:00.074) 0:01:29.467 ******* 2019-12-09T20:26:42.740 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-09T20:26:42.742 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-12-09T20:26:43.068 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-12-09T20:26:43.101 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-12-09T20:26:43.405 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-12-09T20:26:43.434 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-12-09T20:26:43.752 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-12-09T20:26:43.780 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-12-09T20:26:44.087 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-12-09T20:26:44.141 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-12-09T20:26:44.442 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-12-09T20:26:44.456 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-12-09T20:26:44.799 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2019-12-09T20:26:44.801 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-12-09T20:26:45.154 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} ok: [smithi081.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-12-09T20:26:45.477 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-12-09T20:26:45.488 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-12-09T20:26:45.813 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-12-09T20:26:45.839 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-12-09T20:26:46.124 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-12-09T20:26:46.162 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-12-09T20:26:46.444 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-12-09T20:26:46.498 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-12-09T20:26:46.751 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-12-09T20:26:46.828 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-12-09T20:26:47.080 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-12-09T20:26:47.174 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-12-09T20:26:47.413 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-12-09T20:26:47.467 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-12-09T20:26:47.735 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-12-09T20:26:47.790 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-12-09T20:26:48.092 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-12-09T20:26:48.130 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-12-09T20:26:48.437 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-12-09T20:26:48.484 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-12-09T20:26:48.773 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-12-09T20:26:48.817 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-12-09T20:26:49.130 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-12-09T20:26:49.164 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-12-09T20:26:49.484 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-12-09T20:26:49.489 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-12-09T20:26:49.842 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-12-09T20:26:50.034 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jj", "name": "jj", "remove": false, "state": "absent"} 2019-12-09T20:26:50.488 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jj", "name": "jj", "remove": false, "state": "absent"} 2019-12-09T20:26:50.602 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "amarangone", "name": "amarangone", "remove": false, "state": "absent"} 2019-12-09T20:26:51.077 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "amarangone", "name": "amarangone", "remove": false, "state": "absent"} 2019-12-09T20:26:51.190 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "oprypin", "name": "oprypin", "remove": false, "state": "absent"} 2019-12-09T20:26:51.652 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "oprypin", "name": "oprypin", "remove": false, "state": "absent"} 2019-12-09T20:26:51.789 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adamyanova", "name": "adamyanova", "remove": false, "state": "absent"} 2019-12-09T20:26:52.155 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adamyanova", "name": "adamyanova", "remove": false, "state": "absent"} 2019-12-09T20:26:52.315 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sbillah", "name": "sbillah", "remove": false, "state": "absent"} 2019-12-09T20:26:52.748 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sbillah", "name": "sbillah", "remove": false, "state": "absent"} 2019-12-09T20:26:52.896 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "onyb", "name": "onyb", "remove": false, "state": "absent"} 2019-12-09T20:26:53.330 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "onyb", "name": "onyb", "remove": false, "state": "absent"} 2019-12-09T20:26:53.498 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jwilliamson", "name": "jwilliamson", "remove": false, "state": "absent"} 2019-12-09T20:26:53.890 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jwilliamson", "name": "jwilliamson", "remove": false, "state": "absent"} 2019-12-09T20:26:54.091 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "kmroz", "name": "kmroz", "remove": false, "state": "absent"} 2019-12-09T20:26:54.426 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "kmroz", "name": "kmroz", "remove": false, "state": "absent"} 2019-12-09T20:26:54.661 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "shehbazj", "name": "shehbazj", "remove": false, "state": "absent"} 2019-12-09T20:26:55.020 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "shehbazj", "name": "shehbazj", "remove": false, "state": "absent"} 2019-12-09T20:26:55.171 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "remove": false, "state": "absent"} 2019-12-09T20:26:55.591 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "remove": false, "state": "absent"} 2019-12-09T20:26:55.750 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "asheplyakov", "name": "asheplyakov", "remove": false, "state": "absent"} 2019-12-09T20:26:56.140 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "asheplyakov", "name": "asheplyakov", "remove": false, "state": "absent"} 2019-12-09T20:26:56.361 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "liupan", "name": "liupan", "remove": false, "state": "absent"} 2019-12-09T20:26:56.707 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-12-09T20:26:56.760 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "liupan", "name": "liupan", "remove": false, "state": "absent"} 2019-12-09T20:26:57.034 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-12-09T20:26:57.086 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-12-09T20:26:57.340 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-12-09T20:26:57.415 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-12-09T20:26:57.682 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-12-09T20:26:57.780 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-12-09T20:26:57.991 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-12-09T20:26:58.118 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-12-09T20:26:58.348 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-12-09T20:26:58.464 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-12-09T20:26:58.686 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-12-09T20:26:58.775 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-12-09T20:26:59.076 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-12-09T20:26:59.228 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "fche", "name": "fche", "remove": false, "state": "absent"} 2019-12-09T20:26:59.308 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:59 +0000 (0:00:16.623) 0:01:46.090 ******* 2019-12-09T20:26:59.715 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "fche", "name": "fche", "remove": false, "state": "absent"} 2019-12-09T20:26:59.775 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:26:59 +0000 (0:00:00.469) 0:01:46.559 ******* 2019-12-09T20:27:00.438 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2019-12-09T20:27:00.439 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "checksum": "8a303ebcf60c08d57f11c57c9682a02d84685d0b", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c2805c19b557daad4717860e2c7029bc", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575923219.33-129752170820153/source", "state": "file", "uid": 0} 2019-12-09T20:27:00.485 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:00 +0000 (0:00:00.707) 0:01:47.267 ******* 2019-12-09T20:27:00.520 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-12-09T20:27:00.522 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => { "msg": "Host smithi067.front.sepia.ceph.com is running CentOS 7.6 (Core)" } 2019-12-09T20:27:00.560 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:00 +0000 (0:00:00.077) 0:01:47.344 ******* 2019-12-09T20:27:00.598 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-12-09T20:27:00.600 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-12-09T20:27:00.654 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:00 +0000 (0:00:00.094) 0:01:47.438 ******* 2019-12-09T20:27:00.702 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-12-09T20:27:00.704 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:00.748 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:00 +0000 (0:00:00.090) 0:01:47.529 ******* 2019-12-09T20:27:00.862 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2019-12-09T20:27:00.863 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "checksum": "8a303ebcf60c08d57f11c57c9682a02d84685d0b", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "c2805c19b557daad4717860e2c7029bc", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575923219.8-32656607913844/source", "state": "file", "uid": 0} 2019-12-09T20:27:00.920 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:00 +0000 (0:00:00.174) 0:01:47.704 ******* 2019-12-09T20:27:00.958 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-12-09T20:27:00.960 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => { "msg": "Host smithi081.front.sepia.ceph.com is running CentOS 7.6 (Core)" } 2019-12-09T20:27:01.010 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.089) 0:01:47.794 ******* 2019-12-09T20:27:01.068 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2019-12-09T20:27:01.070 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-12-09T20:27:01.122 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.112) 0:01:47.907 ******* 2019-12-09T20:27:01.163 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-12-09T20:27:01.164 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.200 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.077) 0:01:47.984 ******* 2019-12-09T20:27:01.206 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-12-09T20:27:01.207 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.089811", "end": "2019-12-09 20:27:01.151087", "rc": 0, "start": "2019-12-09 20:27:01.061276", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-12-09T20:27:01.247 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.046) 0:01:48.031 ******* 2019-12-09T20:27:01.287 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-12-09T20:27:01.289 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.334 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.082) 0:01:48.114 ******* 2019-12-09T20:27:01.381 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-12-09T20:27:01.383 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.425 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.095) 0:01:48.209 ******* 2019-12-09T20:27:01.474 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-12-09T20:27:01.474 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.523 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.097) 0:01:48.307 ******* 2019-12-09T20:27:01.565 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-12-09T20:27:01.566 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.609 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.085) 0:01:48.393 ******* 2019-12-09T20:27:01.615 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-12-09T20:27:01.617 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.089150", "end": "2019-12-09 20:27:01.579333", "rc": 0, "start": "2019-12-09 20:27:01.490183", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-12-09T20:27:01.659 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.049) 0:01:48.443 ******* 2019-12-09T20:27:01.666 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2019-12-09T20:27:01.667 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.706 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.047) 0:01:48.490 ******* 2019-12-09T20:27:01.711 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-12-09T20:27:01.713 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.753 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.047) 0:01:48.537 ******* 2019-12-09T20:27:01.765 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-12-09T20:27:01.766 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.802 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.048) 0:01:48.586 ******* 2019-12-09T20:27:01.810 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.851 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.046) 0:01:48.633 ******* 2019-12-09T20:27:01.854 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-12-09T20:27:01.856 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi067.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:01.904 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.049) 0:01:48.682 ******* TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-12-09T20:27:01.906 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.944 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.046) 0:01:48.728 ******* 2019-12-09T20:27:01.954 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-12-09T20:27:01.956 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:01.989 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:01 +0000 (0:00:00.045) 0:01:48.773 ******* 2019-12-09T20:27:01.996 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.034 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.044) 0:01:48.818 ******* 2019-12-09T20:27:02.040 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-12-09T20:27:02.042 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.094 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.053) 0:01:48.871 ******* TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.135 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.048) 0:01:48.919 ******* 2019-12-09T20:27:02.141 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-12-09T20:27:02.142 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:02.182 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.046) 0:01:48.966 ******* 2019-12-09T20:27:02.190 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-12-09T20:27:02.192 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.230 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.048) 0:01:49.014 ******* 2019-12-09T20:27:02.237 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-12-09T20:27:02.237 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.278 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.048) 0:01:49.063 ******* 2019-12-09T20:27:02.287 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi081.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi081.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:02.327 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.048) 0:01:49.111 ******* 2019-12-09T20:27:02.333 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-12-09T20:27:02.335 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:02.373 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.045) 0:01:49.157 ******* 2019-12-09T20:27:02.379 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-12-09T20:27:02.381 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.432 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.059) 0:01:49.217 ******* 2019-12-09T20:27:02.443 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.483 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.049) 0:01:49.266 ******* 2019-12-09T20:27:02.490 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-12-09T20:27:02.492 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.530 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.047) 0:01:49.314 ******* 2019-12-09T20:27:02.536 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-12-09T20:27:02.537 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.579 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.049) 0:01:49.363 ******* 2019-12-09T20:27:02.586 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-12-09T20:27:02.587 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:02.628 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.048) 0:01:49.412 ******* 2019-12-09T20:27:02.633 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.677 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.049) 0:01:49.461 ******* 2019-12-09T20:27:02.683 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-12-09T20:27:02.685 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.727 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.049) 0:01:49.511 ******* 2019-12-09T20:27:02.732 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-12-09T20:27:02.734 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.771 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.044) 0:01:49.555 ******* 2019-12-09T20:27:02.778 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-12-09T20:27:02.780 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:02.816 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.044) 0:01:49.600 ******* 2019-12-09T20:27:02.821 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-12-09T20:27:02.823 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:02.865 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.048) 0:01:49.649 ******* 2019-12-09T20:27:02.876 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-12-09T20:27:02.878 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.920 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.055) 0:01:49.704 ******* 2019-12-09T20:27:02.926 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-12-09T20:27:02.936 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:02.968 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:02 +0000 (0:00:00.047) 0:01:49.752 ******* 2019-12-09T20:27:02.976 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2019-12-09T20:27:02.977 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.016 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.048) 0:01:49.800 ******* 2019-12-09T20:27:03.023 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-12-09T20:27:03.024 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.082 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.065) 0:01:49.866 ******* 2019-12-09T20:27:03.089 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-12-09T20:27:03.091 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.141 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.059) 0:01:49.925 ******* 2019-12-09T20:27:03.147 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-12-09T20:27:03.149 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.198 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.054) 0:01:49.980 ******* 2019-12-09T20:27:03.206 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-12-09T20:27:03.208 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.248 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.048) 0:01:50.029 ******* 2019-12-09T20:27:03.257 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.294 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.048) 0:01:50.078 ******* 2019-12-09T20:27:03.300 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi081.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:03.341 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.047) 0:01:50.125 ******* 2019-12-09T20:27:03.347 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-12-09T20:27:03.348 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.387 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.045) 0:01:50.171 ******* 2019-12-09T20:27:03.393 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-12-09T20:27:03.394 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.435 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.048) 0:01:50.219 ******* 2019-12-09T20:27:03.437 INFO:teuthology.task.ansible.out: 2019-12-09T20:27:03.444 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-12-09T20:27:03.445 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.483 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.047) 0:01:50.267 ******* 2019-12-09T20:27:03.491 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-12-09T20:27:03.492 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.532 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.048) 0:01:50.316 ******* 2019-12-09T20:27:03.539 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi067.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:03.578 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.046) 0:01:50.362 ******* 2019-12-09T20:27:03.584 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2019-12-09T20:27:03.586 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.626 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.048) 0:01:50.410 ******* 2019-12-09T20:27:03.633 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-12-09T20:27:03.673 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.046) 0:01:50.457 ******* 2019-12-09T20:27:03.681 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-12-09T20:27:03.689 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.726 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.049) 0:01:50.507 ******* 2019-12-09T20:27:03.730 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.775 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.051) 0:01:50.559 ******* 2019-12-09T20:27:03.783 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2019-12-09T20:27:03.784 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.824 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.049) 0:01:50.608 ******* 2019-12-09T20:27:03.853 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-12-09T20:27:03.854 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:03.893 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.069) 0:01:50.677 ******* 2019-12-09T20:27:03.942 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-12-09T20:27:03.944 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-09T20:27:03.980 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:03 +0000 (0:00:00.087) 0:01:50.764 ******* 2019-12-09T20:27:04.031 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-12-09T20:27:04.077 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:04 +0000 (0:00:00.096) 0:01:50.861 ******* 2019-12-09T20:27:04.094 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-12-09T20:27:04.094 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-09T20:27:04.136 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:04 +0000 (0:00:00.058) 0:01:50.919 ******* 2019-12-09T20:27:04.142 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-12-09T20:27:04.144 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:04.206 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:04 +0000 (0:00:00.070) 0:01:50.990 ******* 2019-12-09T20:27:04.553 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi081.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-09T20:27:04.600 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:04 +0000 (0:00:00.394) 0:01:51.384 ******* 2019-12-09T20:27:05.621 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi067.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-12-09T20:27:05.655 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-12-09T20:27:06.581 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-12-09T20:27:06.632 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:06 +0000 (0:00:02.032) 0:01:53.417 ******* 2019-12-09T20:27:06.641 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-12-09T20:27:06.696 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:06 +0000 (0:00:00.062) 0:01:53.479 ******* 2019-12-09T20:27:07.845 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-12-09T20:27:07.847 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "md5sum": "6467980f3cbb5aa63a3cfe239626f53e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575923226.67-160072983251607/source", "state": "file", "uid": 0} 2019-12-09T20:27:07.900 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "md5sum": "6467980f3cbb5aa63a3cfe239626f53e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575923226.74-78123377576568/source", "state": "file", "uid": 0} 2019-12-09T20:27:09.009 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "md5sum": "0848bb8607c5b0c5706d8e1930fa7d55", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575923227.91-205194646932266/source", "state": "file", "uid": 0} 2019-12-09T20:27:09.049 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:09 +0000 (0:00:02.353) 0:01:55.833 ******* 2019-12-09T20:27:09.056 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "md5sum": "0848bb8607c5b0c5706d8e1930fa7d55", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575923227.86-212352985377881/source", "state": "file", "uid": 0} 2019-12-09T20:27:09.103 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:09 +0000 (0:00:00.054) 0:01:55.887 ******* 2019-12-09T20:27:09.708 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2019-12-09T20:27:09.709 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made", "warnings": ["Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually"]} 2019-12-09T20:27:09.731 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made", "warnings": ["Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually"]} 2019-12-09T20:27:10.010 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-09T20:27:10.014 INFO:teuthology.task.ansible.out: [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually 2019-12-09T20:27:10.067 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:10 +0000 (0:00:00.962) 0:01:56.850 ******* 2019-12-09T20:27:10.078 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-09T20:27:10.128 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:10 +0000 (0:00:00.060) 0:01:56.911 ******* 2019-12-09T20:27:22.504 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** 2019-12-09T20:27:22.504 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2019-12-09T20:27:22.506 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rf /var/cache/yum/*; yum clean all; yum makecache", "delta": "0:00:12.023782", "end": "2019-12-09 20:27:22.476163", "rc": 0, "start": "2019-12-09 20:27:10.452381", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nLoaded plugins: fastestmirror, langpacks, priorities\nDetermining fastest mirrors\n * base: mirror.ash.fastserv.com\n * extras: mirror.math.princeton.edu\n * updates: mirror.linux.duke.edu\nMetadata Cache Created", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Loaded plugins: fastestmirror, langpacks, priorities", "Determining fastest mirrors", " * base: mirror.ash.fastserv.com", " * extras: mirror.math.princeton.edu", " * updates: mirror.linux.duke.edu", "Metadata Cache Created"]} 2019-12-09T20:27:22.543 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:22 +0000 (0:00:12.416) 0:02:09.327 ******* 2019-12-09T20:27:22.899 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* 2019-12-09T20:27:22.900 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made"} 2019-12-09T20:27:23.208 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-09T20:27:23.261 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.717) 0:02:10.045 ******* 2019-12-09T20:27:23.290 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:23.328 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.066) 0:02:10.112 ******* 2019-12-09T20:27:23.359 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-12-09T20:27:23.359 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:23.410 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.082) 0:02:10.195 ******* 2019-12-09T20:27:23.441 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-12-09T20:27:23.443 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:23.491 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.080) 0:02:10.275 ******* 2019-12-09T20:27:23.521 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-09T20:27:23.523 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:23.575 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.084) 0:02:10.359 ******* 2019-12-09T20:27:23.577 INFO:teuthology.task.ansible.out: 2019-12-09T20:27:23.584 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** 2019-12-09T20:27:23.586 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rf /var/cache/yum/*; yum clean all; yum makecache", "delta": "0:00:13.142117", "end": "2019-12-09 20:27:23.509437", "rc": 0, "start": "2019-12-09 20:27:10.367320", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nLoaded plugins: fastestmirror, langpacks, priorities\nDetermining fastest mirrors\n * base: mirror.ash.fastserv.com\n * extras: mirror.math.princeton.edu\n * updates: mirror.linux.duke.edu\nMetadata Cache Created", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Loaded plugins: fastestmirror, langpacks, priorities", "Determining fastest mirrors", " * base: mirror.ash.fastserv.com", " * extras: mirror.math.princeton.edu", " * updates: mirror.linux.duke.edu", "Metadata Cache Created"]} 2019-12-09T20:27:23.633 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.056) 0:02:10.416 ******* 2019-12-09T20:27:23.639 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-12-09T20:27:23.639 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:23.688 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.057) 0:02:10.473 ******* 2019-12-09T20:27:23.717 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-12-09T20:27:23.718 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:23.755 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.065) 0:02:10.539 ******* 2019-12-09T20:27:23.791 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-12-09T20:27:23.793 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:23.837 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.082) 0:02:10.621 ******* 2019-12-09T20:27:23.864 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-12-09T20:27:23.866 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:23.907 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:23 +0000 (0:00:00.070) 0:02:10.691 ******* 2019-12-09T20:27:24.008 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* 2019-12-09T20:27:24.009 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made"} 2019-12-09T20:27:24.320 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-09T20:27:24.366 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:24 +0000 (0:00:00.458) 0:02:11.150 ******* 2019-12-09T20:27:24.396 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-09T20:27:24.397 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:24.458 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:24 +0000 (0:00:00.092) 0:02:11.242 ******* 2019-12-09T20:27:24.485 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-12-09T20:27:24.486 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:24.526 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:24 +0000 (0:00:00.067) 0:02:11.310 ******* 2019-12-09T20:27:24.557 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-12-09T20:27:24.558 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:24.600 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:24 +0000 (0:00:00.074) 0:02:11.384 ******* 2019-12-09T20:27:24.627 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-09T20:27:24.629 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:24.679 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:24 +0000 (0:00:00.079) 0:02:11.463 ******* 2019-12-09T20:27:24.706 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-12-09T20:27:24.707 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:24.760 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:24 +0000 (0:00:00.080) 0:02:11.544 ******* 2019-12-09T20:27:24.793 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-12-09T20:27:24.794 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:24.848 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:24 +0000 (0:00:00.087) 0:02:11.631 ******* 2019-12-09T20:27:24.874 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:24.923 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:24 +0000 (0:00:00.075) 0:02:11.707 ******* 2019-12-09T20:27:24.951 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-12-09T20:27:24.952 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:24.996 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:24 +0000 (0:00:00.072) 0:02:11.780 ******* 2019-12-09T20:27:25.530 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-12-09T20:27:25.537 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.299887", "end": "2019-12-09 20:27:25.500450", "rc": 0, "start": "2019-12-09 20:27:24.200563", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:27:25.577 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:25 +0000 (0:00:00.580) 0:02:12.361 ******* 2019-12-09T20:27:26.445 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2019-12-09T20:27:26.485 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:26 +0000 (0:00:00.908) 0:02:13.269 ******* 2019-12-09T20:27:26.516 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-12-09T20:27:26.518 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:26.560 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:26 +0000 (0:00:00.074) 0:02:13.344 ******* 2019-12-09T20:27:26.566 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-12-09T20:27:26.567 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.202147", "end": "2019-12-09 20:27:26.500443", "rc": 0, "start": "2019-12-09 20:27:25.298296", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:27:26.613 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:26 +0000 (0:00:00.053) 0:02:13.398 ******* 2019-12-09T20:27:26.622 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-12-09T20:27:26.624 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:26.668 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:26 +0000 (0:00:00.054) 0:02:13.452 ******* 2019-12-09T20:27:26.694 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2019-12-09T20:27:26.696 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:26.741 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:26 +0000 (0:00:00.068) 0:02:13.521 ******* 2019-12-09T20:27:26.888 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-12-09T20:27:26.890 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:26.928 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:26 +0000 (0:00:00.190) 0:02:13.712 ******* 2019-12-09T20:27:27.126 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-12-09T20:27:27.127 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2019-12-09T20:27:27.164 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:27 +0000 (0:00:00.234) 0:02:13.946 ******* 2019-12-09T20:27:27.190 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-12-09T20:27:27.192 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:27.230 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:27 +0000 (0:00:00.067) 0:02:14.014 ******* 2019-12-09T20:27:27.256 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-12-09T20:27:27.258 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:27.305 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:27 +0000 (0:00:00.074) 0:02:14.089 ******* 2019-12-09T20:27:27.332 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2019-12-09T20:27:27.334 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:27.373 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:27 +0000 (0:00:00.068) 0:02:14.157 ******* 2019-12-09T20:27:27.400 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-12-09T20:27:27.402 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:27.446 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:27 +0000 (0:00:00.072) 0:02:14.230 ******* 2019-12-09T20:27:27.984 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-12-09T20:27:27.991 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-12-09T20:27:28.059 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:28 +0000 (0:00:00.612) 0:02:14.843 ******* 2019-12-09T20:27:28.315 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-12-09T20:27:28.355 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:28 +0000 (0:00:00.296) 0:02:15.139 ******* 2019-12-09T20:27:31.847 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2019-12-09T20:27:31.848 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2019-12-09T20:27:31.886 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:31 +0000 (0:00:03.530) 0:02:18.670 ******* 2019-12-09T20:27:32.268 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2019-12-09T20:27:32.340 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:32 +0000 (0:00:00.454) 0:02:19.124 ******* 2019-12-09T20:27:32.833 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-12-09T20:27:32.834 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-12-09T20:27:33.241 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-12-09T20:27:33.698 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-12-09T20:27:34.123 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-12-09T20:27:34.620 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-12-09T20:27:34.666 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:34 +0000 (0:00:02.325) 0:02:21.450 ******* 2019-12-09T20:27:34.984 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-12-09T20:27:34.986 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-09T20:27:35.034 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:35 +0000 (0:00:00.366) 0:02:21.816 ******* 2019-12-09T20:27:35.038 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-12-09T20:27:35.039 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-12-09T20:27:35.080 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:35 +0000 (0:00:00.047) 0:02:21.864 ******* 2019-12-09T20:27:35.411 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-12-09T20:27:35.412 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-09T20:27:35.458 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:35 +0000 (0:00:00.373) 0:02:22.238 ******* 2019-12-09T20:27:35.950 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-09T20:27:35.953 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-12-09T20:27:36.318 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-12-09T20:27:36.900 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-12-09T20:27:37.172 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-12-09T20:27:37.768 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-12-09T20:27:37.808 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:37 +0000 (0:00:02.354) 0:02:24.592 ******* 2019-12-09T20:27:37.835 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-12-09T20:27:37.836 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:37.872 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:37 +0000 (0:00:00.064) 0:02:24.656 ******* 2019-12-09T20:27:37.910 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-12-09T20:27:37.912 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-12-09T20:27:37.950 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:37 +0000 (0:00:00.078) 0:02:24.734 ******* 2019-12-09T20:27:38.061 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-09T20:27:38.062 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-12-09T20:27:38.107 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:38 +0000 (0:00:00.156) 0:02:24.891 ******* 2019-12-09T20:27:38.144 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:38.189 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:38 +0000 (0:00:00.082) 0:02:24.974 ******* 2019-12-09T20:27:38.229 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-12-09T20:27:38.230 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-12-09T20:27:38.280 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:38 +0000 (0:00:00.090) 0:02:25.064 ******* 2019-12-09T20:27:38.286 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-12-09T20:27:38.288 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-09T20:27:38.328 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:38 +0000 (0:00:00.047) 0:02:25.112 ******* 2019-12-09T20:27:38.374 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-12-09T20:27:38.375 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-09T20:27:38.385 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-09T20:27:38.406 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-09T20:27:38.427 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-09T20:27:38.465 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:38 +0000 (0:00:00.137) 0:02:25.249 ******* 2019-12-09T20:27:38.620 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-12-09T20:27:38.621 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-09T20:27:38.658 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:38 +0000 (0:00:00.193) 0:02:25.442 ******* 2019-12-09T20:27:38.707 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-12-09T20:27:38.708 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-09T20:27:38.740 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-09T20:27:38.742 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-09T20:27:38.751 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-09T20:27:38.788 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:38 +0000 (0:00:00.130) 0:02:25.573 ******* 2019-12-09T20:27:41.905 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-12-09T20:27:41.906 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2019-12-09T20:27:41.956 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:41 +0000 (0:00:03.167) 0:02:28.740 ******* 2019-12-09T20:27:41.985 INFO:teuthology.task.ansible.out: 2019-12-09T20:27:41.987 INFO:teuthology.task.ansible.out:TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:42.025 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:42 +0000 (0:00:00.069) 0:02:28.810 ******* 2019-12-09T20:27:42.074 INFO:teuthology.task.ansible.out: 2019-12-09T20:27:42.076 INFO:teuthology.task.ansible.out:TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2019-12-09T20:27:42.122 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:42 +0000 (0:00:00.095) 0:02:28.906 ******* 2019-12-09T20:27:42.150 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:42.193 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:42 +0000 (0:00:00.071) 0:02:28.977 ******* 2019-12-09T20:27:42.983 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-12-09T20:27:42.984 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-12-09T20:27:43.028 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:43 +0000 (0:00:00.835) 0:02:29.812 ******* 2019-12-09T20:27:43.127 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-12-09T20:27:43.179 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:43 +0000 (0:00:00.150) 0:02:29.963 ******* 2019-12-09T20:27:44.036 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-12-09T20:27:44.081 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:44 +0000 (0:00:00.902) 0:02:30.865 ******* 2019-12-09T20:27:44.090 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-12-09T20:27:44.092 INFO:teuthology.task.ansible.out: 2019-12-09T20:27:44.142 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:44 +0000 (0:00:00.060) 0:02:30.926 ******* 2019-12-09T20:27:44.148 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-12-09T20:27:44.148 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:44.191 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:44 +0000 (0:00:00.049) 0:02:30.975 ******* 2019-12-09T20:27:44.204 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:44.245 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:44 +0000 (0:00:00.053) 0:02:31.029 ******* 2019-12-09T20:27:44.515 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi067.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-09T20:27:44.552 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:44 +0000 (0:00:00.307) 0:02:31.337 ******* 2019-12-09T20:27:44.554 INFO:teuthology.task.ansible.out: 2019-12-09T20:27:44.591 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-09T20:27:44.634 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:44 +0000 (0:00:00.081) 0:02:31.418 ******* 2019-12-09T20:27:44.918 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-12-09T20:27:44.919 INFO:teuthology.task.ansible.out:fatal: [smithi067.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-12-09T20:27:44.970 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:44 +0000 (0:00:00.335) 0:02:31.754 ******* 2019-12-09T20:27:45.009 INFO:teuthology.task.ansible.out:fatal: [smithi081.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-09T20:27:45.014 INFO:teuthology.task.ansible.out:...ignoring TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:45.064 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:45 +0000 (0:00:00.094) 0:02:31.848 ******* 2019-12-09T20:27:45.103 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:45 +0000 (0:00:00.038) 0:02:31.887 ******* 2019-12-09T20:27:45.154 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:27:45.211 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:45 +0000 (0:00:00.108) 0:02:31.995 ******* 2019-12-09T20:27:46.013 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-12-09T20:27:46.058 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:46 +0000 (0:00:00.843) 0:02:32.839 ******* 2019-12-09T20:27:46.161 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-12-09T20:27:46.201 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:46 +0000 (0:00:00.146) 0:02:32.985 ******* 2019-12-09T20:27:47.085 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-12-09T20:27:47.088 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2019-12-09 20:24:32 UTC", "ActiveEnterTimestampMonotonic": "19504955", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket nss-lookup.target var-run.mount time-sync.target local-fs.target basic.target network.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Mon 2019-12-09 20:24:32 UTC", "AssertTimestampMonotonic": "19503555", "Before": "shutdown.target getty@tty1.service xdm.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2019-12-09 20:24:32 UTC", "ConditionTimestampMonotonic": "19503554", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "8940", "ExecMainStartTimestamp": "Mon 2019-12-09 20:24:32 UTC", "ExecMainStartTimestampMonotonic": "19504859", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_err 2019-12-09T20:27:47.090 INFO:teuthology.task.ansible.out:ors=no ; start_time=[Mon 2019-12-09 20:24:32 UTC] ; stop_time=[n/a] ; pid=8940 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2019-12-09 20:24:32 UTC", "InactiveExitTimestampMonotonic": "19504955", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "8940", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target network-online.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOn 2019-12-09T20:27:47.091 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Mon 2019-12-09 20:24:32 UTC", "WatchdogTimestampMonotonic": "19504919", "WatchdogUSec": "0"}} 2019-12-09T20:27:47.138 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:47 +0000 (0:00:00.936) 0:02:33.922 ******* 2019-12-09T20:27:47.149 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2019-12-09 20:24:23 UTC", "ActiveEnterTimestampMonotonic": "21802880", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target var-run.mount time-sync.target system.slice basic.target local-fs.target network.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Mon 2019-12-09 20:24:23 UTC", "AssertTimestampMonotonic": "21802367", "Before": "getty@tty1.service shutdown.target multi-user.target xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2019-12-09 20:24:23 UTC", "ConditionTimestampMonotonic": "21802365", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "8940", "ExecMainStartTimestamp": "Mon 2019-12-09 20:24:23 UTC", "ExecMainStartTimestampMonotonic": "21802833", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_err 2019-12-09T20:27:47.151 INFO:teuthology.task.ansible.out:ors=no ; start_time=[Mon 2019-12-09 20:24:23 UTC] ; stop_time=[n/a] ; pid=8940 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2019-12-09 20:24:23 UTC", "InactiveExitTimestampMonotonic": "21802880", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "8940", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "network-online.target basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOn 2019-12-09T20:27:47.160 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Mon 2019-12-09 20:24:23 UTC", "WatchdogTimestampMonotonic": "21802862", "WatchdogUSec": "0"}} 2019-12-09T20:27:47.198 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:47 +0000 (0:00:00.059) 0:02:33.982 ******* 2019-12-09T20:27:47.468 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-12-09T20:27:47.470 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.004903", "end": "2019-12-09 20:27:47.443401", "rc": 0, "start": "2019-12-09 20:27:47.438498", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-12-09T20:27:47.507 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:47 +0000 (0:00:00.307) 0:02:34.289 ******* 2019-12-09T20:27:47.534 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.004738", "end": "2019-12-09 20:27:47.506824", "rc": 0, "start": "2019-12-09 20:27:47.502086", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-12-09T20:27:47.572 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:47 +0000 (0:00:00.066) 0:02:34.356 ******* 2019-12-09T20:27:48.029 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-12-09T20:27:48.031 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-12-09T20:27:48.086 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:48 +0000 (0:00:00.513) 0:02:34.870 ******* 2019-12-09T20:27:48.094 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-12-09T20:27:48.148 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:27:48 +0000 (0:00:00.061) 0:02:34.931 ******* 2019-12-09T20:28:15.393 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["selinux-policy-targeted", "3.13.1-252.el7_7.6.noarch from updates"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.ash.fastserv.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.math.princeton.edu\n * updates: mirror.linux.duke.edu\nResolving Dependencies\n--> Running transaction check\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Running transaction check\n---> Package selinux-policy.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n selinux-policy-targeted noarch 3.13.1-252.el7_7.6 updates 7.0 M\nUpdating for dependencies:\n selinux-policy noarch 3.13.1-252.el7_7.6 updates 492 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTotal download size: 7.5 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n---------------------------- 2019-12-09T20:28:15.395 INFO:teuthology.task.ansible.out:----------------------------------------------------\nTotal 15 MB/s | 7.5 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Updating : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Cleanup : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Cleanup : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n Verifying : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Verifying : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n\nUpdated:\n selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 \n\nDependency Updated:\n selinux-policy.noarch 0:3.13.1-252.el7_7.6 \n\nComplete!\n"]} 2019-12-09T20:28:15.429 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:15 +0000 (0:00:27.282) 0:03:02.214 ******* 2019-12-09T20:28:15.791 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["selinux-policy-targeted", "3.13.1-252.el7_7.6.noarch from updates"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.ash.fastserv.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.math.princeton.edu\n * updates: mirror.linux.duke.edu\nResolving Dependencies\n--> Running transaction check\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Running transaction check\n---> Package selinux-policy.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n selinux-policy-targeted noarch 3.13.1-252.el7_7.6 updates 7.0 M\nUpdating for dependencies:\n selinux-policy noarch 3.13.1-252.el7_7.6 updates 492 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTotal download size: 7.5 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 2019-12-09T20:28:15.793 INFO:teuthology.task.ansible.out: 15 MB/s | 7.5 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Updating : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Cleanup : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Cleanup : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n Verifying : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Verifying : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n\nUpdated:\n selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 \n\nDependency Updated:\n selinux-policy.noarch 0:3.13.1-252.el7_7.6 \n\nComplete!\n"]} 2019-12-09T20:28:15.831 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:15 +0000 (0:00:00.401) 0:03:02.615 ******* 2019-12-09T20:28:16.110 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-12-09T20:28:16.112 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-12-09T20:28:16.155 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:16 +0000 (0:00:00.324) 0:03:02.939 ******* 2019-12-09T20:28:16.434 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-12-09T20:28:16.475 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:16 +0000 (0:00:00.319) 0:03:03.259 ******* 2019-12-09T20:28:16.476 INFO:teuthology.task.ansible.out: 2019-12-09T20:28:24.278 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-12-09T20:28:24.279 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:07.818897", "end": "2019-12-09 20:28:24.243758", "failed_when_result": false, "rc": 0, "start": "2019-12-09 20:28:16.424861", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-12-09T20:28:24.351 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:24 +0000 (0:00:07.876) 0:03:11.135 ******* 2019-12-09T20:28:25.180 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:08.366697", "end": "2019-12-09 20:28:25.140295", "failed_when_result": false, "rc": 0, "start": "2019-12-09 20:28:16.773598", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-12-09T20:28:25.219 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:25 +0000 (0:00:00.867) 0:03:12.003 ******* 2019-12-09T20:28:25.278 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-12-09T20:28:25.279 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2019-12-09T20:28:25.328 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:25 +0000 (0:00:00.109) 0:03:12.112 ******* 2019-12-09T20:28:25.674 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-12-09T20:28:25.675 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.018487", "end": "2019-12-09 20:28:25.647777", "rc": 0, "start": "2019-12-09 20:28:25.629290", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-12-09T20:28:25.715 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:25 +0000 (0:00:00.383) 0:03:12.496 ******* 2019-12-09T20:28:26.038 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.015598", "end": "2019-12-09 20:28:26.011743", "rc": 0, "start": "2019-12-09 20:28:25.996145", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:28:26.090 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:26 +0000 (0:00:00.377) 0:03:12.874 ******* 2019-12-09T20:28:26.180 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2019-12-09T20:28:26.231 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:26 +0000 (0:00:00.140) 0:03:13.015 ******* 2019-12-09T20:28:26.580 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-12-09T20:28:26.581 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.017390", "end": "2019-12-09 20:28:26.551595", "rc": 0, "start": "2019-12-09 20:28:26.534205", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-12-09T20:28:26.631 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:26 +0000 (0:00:00.400) 0:03:13.415 ******* 2019-12-09T20:28:26.975 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-12-09T20:28:26.977 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.016153", "end": "2019-12-09 20:28:26.952105", "rc": 0, "start": "2019-12-09 20:28:26.935952", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:28:27.017 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:27 +0000 (0:00:00.385) 0:03:13.801 ******* 2019-12-09T20:28:34.279 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-12-09T20:28:34.281 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:07.883291", "end": "2019-12-09 20:28:34.244121", "rc": 0, "start": "2019-12-09 20:28:26.360830", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:28:34.327 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:34 +0000 (0:00:07.310) 0:03:21.111 ******* 2019-12-09T20:28:34.681 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-12-09T20:28:34.683 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-12-09T20:28:34.725 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:34 +0000 (0:00:00.398) 0:03:21.509 ******* 2019-12-09T20:28:34.754 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-12-09T20:28:34.756 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-12-09T20:28:34.792 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:34 +0000 (0:00:00.064) 0:03:21.573 ******* 2019-12-09T20:28:34.836 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-12-09T20:28:34.838 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-09T20:28:34.876 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:34 +0000 (0:00:00.085) 0:03:21.659 ******* 2019-12-09T20:28:34.923 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-12-09T20:28:34.924 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "podman", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-12-09T20:28:34.975 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:34 +0000 (0:00:00.098) 0:03:21.758 ******* 2019-12-09T20:28:35.016 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-12-09T20:28:35.018 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-09T20:28:35.056 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:35 +0000 (0:00:00.081) 0:03:21.840 ******* 2019-12-09T20:28:35.377 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-12-09T20:28:35.378 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-09T20:28:35.421 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:35 +0000 (0:00:00.361) 0:03:22.201 ******* 2019-12-09T20:28:35.748 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-12-09T20:28:35.750 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2019-12-09T20:28:35.803 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:35 +0000 (0:00:00.386) 0:03:22.588 ******* 2019-12-09T20:28:35.812 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-12-09T20:28:35.812 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:08.418794", "end": "2019-12-09 20:28:35.729014", "rc": 0, "start": "2019-12-09 20:28:27.310220", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:28:35.879 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:35 +0000 (0:00:00.075) 0:03:22.663 ******* 2019-12-09T20:28:35.887 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:35.928 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:35 +0000 (0:00:00.049) 0:03:22.712 ******* 2019-12-09T20:28:35.956 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-12-09T20:28:35.957 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:35.994 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:35 +0000 (0:00:00.065) 0:03:22.778 ******* 2019-12-09T20:28:36.241 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-12-09T20:28:36.242 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-12-09T20:28:36.289 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:36 +0000 (0:00:00.294) 0:03:23.073 ******* 2019-12-09T20:28:36.317 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-12-09T20:28:36.323 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-12-09T20:28:36.365 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:36 +0000 (0:00:00.076) 0:03:23.150 ******* 2019-12-09T20:28:36.372 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-12-09T20:28:36.373 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-12-09T20:28:36.420 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:36 +0000 (0:00:00.053) 0:03:23.203 ******* 2019-12-09T20:28:36.425 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-12-09T20:28:36.425 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-09T20:28:36.466 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:36 +0000 (0:00:00.046) 0:03:23.249 ******* 2019-12-09T20:28:36.520 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi067.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "podman", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-12-09T20:28:36.563 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:36 +0000 (0:00:00.097) 0:03:23.347 ******* 2019-12-09T20:28:36.610 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2019-12-09T20:28:36.611 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-09T20:28:36.659 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:36 +0000 (0:00:00.096) 0:03:23.443 ******* 2019-12-09T20:28:36.984 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-12-09T20:28:36.986 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-09T20:28:37.021 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:37 +0000 (0:00:00.362) 0:03:23.805 ******* 2019-12-09T20:28:37.343 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-12-09T20:28:37.345 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2019-12-09T20:28:37.402 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:37 +0000 (0:00:00.380) 0:03:24.186 ******* 2019-12-09T20:28:37.429 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-12-09T20:28:37.431 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:37.467 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:37 +0000 (0:00:00.065) 0:03:24.251 ******* 2019-12-09T20:28:37.473 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-12-09T20:28:37.474 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-12-09T20:28:37.512 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:37 +0000 (0:00:00.045) 0:03:24.296 ******* 2019-12-09T20:28:37.518 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-12-09T20:28:37.518 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:37.558 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:37 +0000 (0:00:00.046) 0:03:24.342 ******* 2019-12-09T20:28:37.565 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-12-09T20:28:37.566 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi081"}, "changed": false} 2019-12-09T20:28:37.603 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:37 +0000 (0:00:00.044) 0:03:24.387 ******* 2019-12-09T20:28:37.912 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-12-09T20:28:37.913 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-12-09T20:28:37.975 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:37 +0000 (0:00:00.371) 0:03:24.759 ******* 2019-12-09T20:28:38.562 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-12-09T20:28:38.569 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi081.front.sepia.ceph.com", "ansible_hostname": "smithi081", "ansible_nodename": "smithi081"}, "changed": true, "name": "smithi081"} 2019-12-09T20:28:38.624 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:38 +0000 (0:00:00.647) 0:03:25.407 ******* 2019-12-09T20:28:39.082 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2019-12-09T20:28:39.083 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-12-09T20:28:39.139 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:39 +0000 (0:00:00.514) 0:03:25.921 ******* 2019-12-09T20:28:39.187 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-12-09T20:28:39.189 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi067"}, "changed": false} 2019-12-09T20:28:39.233 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:39 +0000 (0:00:00.092) 0:03:26.014 ******* 2019-12-09T20:28:39.558 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2019-12-09T20:28:39.559 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2019-12-09T20:28:39.603 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:39 +0000 (0:00:00.372) 0:03:26.387 ******* 2019-12-09T20:28:39.788 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-12-09T20:28:39.790 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi067.front.sepia.ceph.com", "ansible_hostname": "smithi067", "ansible_nodename": "smithi067"}, "changed": true, "name": "smithi067"} 2019-12-09T20:28:39.827 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:39 +0000 (0:00:00.223) 0:03:26.611 ******* 2019-12-09T20:28:40.548 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-12-09T20:28:40.600 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:40 +0000 (0:00:00.772) 0:03:27.384 ******* 2019-12-09T20:28:40.818 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2019-12-09T20:28:40.866 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:40 +0000 (0:00:00.266) 0:03:27.650 ******* 2019-12-09T20:28:41.546 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-12-09T20:28:41.548 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-12-09T20:28:41.588 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:41 +0000 (0:00:00.721) 0:03:28.372 ******* 2019-12-09T20:28:41.809 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-12-09T20:28:41.811 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-12-09T20:28:41.849 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:41 +0000 (0:00:00.260) 0:03:28.633 ******* 2019-12-09T20:28:42.537 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-12-09T20:28:42.575 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:42 +0000 (0:00:00.725) 0:03:29.359 ******* 2019-12-09T20:28:42.753 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-12-09T20:28:42.754 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-12-09T20:28:42.803 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:42 +0000 (0:00:00.228) 0:03:29.587 ******* 2019-12-09T20:28:42.896 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-09T20:28:42.897 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-09T20:28:42.971 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:42 +0000 (0:00:00.168) 0:03:29.755 ******* 2019-12-09T20:28:43.316 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-09T20:28:43.318 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "gid": 10103, "name": "kvm", "state": "present", "system": false} 2019-12-09T20:28:43.371 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:43 +0000 (0:00:00.399) 0:03:30.155 ******* 2019-12-09T20:28:43.736 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-09T20:28:43.737 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-12-09T20:28:43.782 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:43 +0000 (0:00:00.410) 0:03:30.565 ******* 2019-12-09T20:28:43.796 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-12-09T20:28:43.797 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-12-09T20:28:43.889 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:43 +0000 (0:00:00.107) 0:03:30.673 ******* 2019-12-09T20:28:44.240 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-09T20:28:44.241 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-09T20:28:44.280 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:44 +0000 (0:00:00.390) 0:03:31.064 ******* 2019-12-09T20:28:44.629 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-09T20:28:44.631 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "gid": 10103, "name": "kvm", "state": "present", "system": false} 2019-12-09T20:28:44.687 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:44 +0000 (0:00:00.402) 0:03:31.466 ******* 2019-12-09T20:28:44.775 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-12-09T20:28:44.821 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:44 +0000 (0:00:00.138) 0:03:31.605 ******* 2019-12-09T20:28:45.051 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-09T20:28:45.052 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-12-09T20:28:45.104 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:45 +0000 (0:00:00.283) 0:03:31.888 ******* 2019-12-09T20:28:45.760 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-12-09T20:28:45.824 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:45 +0000 (0:00:00.711) 0:03:32.600 ******* 2019-12-09T20:28:46.093 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-09T20:28:46.095 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-12-09T20:28:46.135 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:46 +0000 (0:00:00.318) 0:03:32.919 ******* 2019-12-09T20:28:47.140 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-12-09T20:28:47.181 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:47 +0000 (0:00:01.046) 0:03:33.965 ******* 2019-12-09T20:28:48.302 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-09T20:28:48.304 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.139182", "end": "2019-12-09 20:28:48.268076", "rc": 0, "start": "2019-12-09 20:28:46.128894", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:28:48.340 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:48 +0000 (0:00:01.159) 0:03:35.124 ******* 2019-12-09T20:28:48.369 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-12-09T20:28:48.410 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:48 +0000 (0:00:00.069) 0:03:35.194 ******* 2019-12-09T20:28:49.361 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-12-09T20:28:49.362 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "state": "file", "uid": 0} 2019-12-09T20:28:49.670 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-09T20:28:49.671 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.138714", "end": "2019-12-09 20:28:49.635801", "rc": 0, "start": "2019-12-09 20:28:47.497087", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:28:49.712 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:49 +0000 (0:00:01.301) 0:03:36.496 ******* 2019-12-09T20:28:49.739 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-12-09T20:28:49.780 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:49 +0000 (0:00:00.068) 0:03:36.564 ******* 2019-12-09T20:28:50.273 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-12-09T20:28:50.275 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "state": "file", "uid": 0} 2019-12-09T20:28:50.332 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:50 +0000 (0:00:00.552) 0:03:37.117 ******* 2019-12-09T20:28:50.361 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-12-09T20:28:50.363 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:50.405 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:50 +0000 (0:00:00.070) 0:03:37.187 ******* 2019-12-09T20:28:50.450 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi081.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-09T20:28:50.492 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:50 +0000 (0:00:00.088) 0:03:37.276 ******* 2019-12-09T20:28:50.519 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-12-09T20:28:50.521 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:50.571 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:50 +0000 (0:00:00.079) 0:03:37.355 ******* 2019-12-09T20:28:50.611 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-09T20:28:50.613 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2019-12-09T20:28:50.671 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:50 +0000 (0:00:00.096) 0:03:37.452 ******* 2019-12-09T20:28:50.840 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** ok: [smithi067.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "state": "file", "uid": 0} 2019-12-09T20:28:51.155 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-12-09T20:28:51.157 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2019-12-09T20:28:51.197 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:51 +0000 (0:00:00.529) 0:03:37.981 ******* 2019-12-09T20:28:51.778 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-12-09T20:28:51.779 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "state": "file", "uid": 0} 2019-12-09T20:28:51.823 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:51 +0000 (0:00:00.625) 0:03:38.607 ******* 2019-12-09T20:28:51.856 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:51.896 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:51 +0000 (0:00:00.073) 0:03:38.680 ******* 2019-12-09T20:28:51.903 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-12-09T20:28:51.905 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.328084", "end": "2019-12-09 20:28:51.831408", "rc": 0, "start": "2019-12-09 20:28:51.503324", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.ash.fastserv.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.math.princeton.edu\n * updates: mirror.linux.duke.edu\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Loading mirror speeds from cached hostfile", " * base: mirror.ash.fastserv.com", " * epel: download-cc-rdu01.fedoraproject.org", " * extras: mirror.math.princeton.edu", " * updates: mirror.linux.duke.edu", "No unfinished transactions left."]} 2019-12-09T20:28:51.949 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:51 +0000 (0:00:00.053) 0:03:38.733 ******* 2019-12-09T20:28:51.955 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-12-09T20:28:51.956 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-09T20:28:51.958 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-09T20:28:51.999 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:51 +0000 (0:00:00.050) 0:03:38.783 ******* 2019-12-09T20:28:52.026 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-12-09T20:28:52.027 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:52.092 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:52 +0000 (0:00:00.092) 0:03:38.876 ******* 2019-12-09T20:28:52.135 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-09T20:28:52.137 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2019-12-09T20:28:52.173 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:52 +0000 (0:00:00.080) 0:03:38.957 ******* 2019-12-09T20:28:52.286 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-12-09T20:28:52.288 INFO:teuthology.task.ansible.out:fatal: [smithi081.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.035809", "end": "2019-12-09 20:28:52.262564", "msg": "non-zero return code", "rc": 1, "start": "2019-12-09 20:28:52.226755", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} 2019-12-09T20:28:52.289 INFO:teuthology.task.ansible.out:...ignoring 2019-12-09T20:28:52.330 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:52 +0000 (0:00:00.157) 0:03:39.114 ******* 2019-12-09T20:28:52.368 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-12-09T20:28:52.370 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:52.413 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:52 +0000 (0:00:00.082) 0:03:39.197 ******* 2019-12-09T20:28:52.671 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-12-09T20:28:52.671 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2019-12-09T20:28:52.732 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:52 +0000 (0:00:00.319) 0:03:39.516 ******* 2019-12-09T20:28:53.380 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-12-09T20:28:53.381 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.304966", "end": "2019-12-09 20:28:53.346658", "rc": 0, "start": "2019-12-09 20:28:53.041692", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.ash.fastserv.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.math.princeton.edu\n * updates: mirror.linux.duke.edu\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Loading mirror speeds from cached hostfile", " * base: mirror.ash.fastserv.com", " * epel: download-cc-rdu01.fedoraproject.org", " * extras: mirror.math.princeton.edu", " * updates: mirror.linux.duke.edu", "No unfinished transactions left."]} 2019-12-09T20:28:53.426 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:53 +0000 (0:00:00.694) 0:03:40.210 ******* 2019-12-09T20:28:53.828 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-12-09T20:28:53.829 INFO:teuthology.task.ansible.out:fatal: [smithi067.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.047373", "end": "2019-12-09 20:28:53.794329", "msg": "non-zero return code", "rc": 1, "start": "2019-12-09 20:28:53.746956", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-12-09T20:28:53.882 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:53 +0000 (0:00:00.456) 0:03:40.667 ******* 2019-12-09T20:28:53.911 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-12-09T20:28:53.913 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:28:53.949 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:28:53 +0000 (0:00:00.066) 0:03:40.733 ******* 2019-12-09T20:29:01.083 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-09T20:29:01.085 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi", "iozone"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n librados2 x86_64 1:10.2.5-4.el7 @base 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 2019-12-09T20:29:01.086 INFO:teuthology.task.ansible.out: @base 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.1 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 @updates 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 4 Packages (+3 Dependent packages)\n\nInstalled size: 23 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 1/7 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 2/7 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 3/7 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/7 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 6/7 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 7/7 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/7 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/7 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/7 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/7 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 7/7 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos 2019-12-09T20:29:01.088 INFO:teuthology.task.ansible.out: \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-09T20:29:01.129 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:01 +0000 (0:00:07.180) 0:03:47.913 ******* 2019-12-09T20:29:02.125 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi", "iozone"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependen 2019-12-09T20:29:02.125 INFO:teuthology.task.ansible.out:cy: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n librados2 x86_64 1:10.2.5-4.el7 @base 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 @base 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.1 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 @updates 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 4 Packages (+3 Dependent packages)\n\nInstalled size: 23 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 1/7 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 2/7 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 3/7 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/7 \n Erasing 2019-12-09T20:29:02.127 INFO:teuthology.task.ansible.out:: 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 6/7 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 7/7 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/7 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/7 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/7 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/7 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 7/7 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-09T20:29:02.165 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:02 +0000 (0:00:01.035) 0:03:48.949 ******* 2019-12-09T20:29:04.503 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "leveldb is not installed", "xmlstarlet is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-12-09T20:29:04.559 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:04 +0000 (0:00:02.389) 0:03:51.339 ******* 2019-12-09T20:29:05.542 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "leveldb is not installed", "xmlstarlet is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-12-09T20:29:05.585 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:05 +0000 (0:00:01.030) 0:03:52.369 ******* 2019-12-09T20:29:22.761 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-09T20:29:22.763 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": ["@core", "@base", "iozone", "qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "msg": "Warning: Group core does not have any packages to install.\nWarning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-114.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-20.el7.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is alrea 2019-12-09T20:29:22.765 INFO:teuthology.task.ansible.out:dy installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed", "gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "parted-3.1-29.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-61.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed", "python-nose-1.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "podman-1.4.4-4.el7.centos.x86_64 providing podman is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-61.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already inst 2019-12-09T20:29:22.767 INFO:teuthology.task.ansible.out:alled", "blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-90.el7.centos.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-90.el7.centos.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-90.el7.centos.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-90.el7.centos.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.65.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed", "python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.ash.fastserv.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.math.princeton.edu\n * updates: mirror.linux.duke.edu\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Proces 2019-12-09T20:29:22.768 INFO:teuthology.task.ansible.out:sing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n librados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 updates 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 updates 435 k\n\nTransaction Summary\n================================================= 2019-12-09T20:29:22.770 INFO:teuthology.task.ansible.out:===============================\nInstall 3 Packages (+5 Dependent packages)\n\nTotal download size: 8.1 M\nInstalled size: 23 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 23 MB/s | 8.1 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/8 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/8 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/8 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/8 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/8 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/8 \n Installing : iozone-3.424-2_ceph.el7.centos.x86_64 7/8 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 8/8 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/8 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/8 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/8 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/8 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/8 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/8 \n Verifying : boost-random-1.53.0-27.el7.x86_64 7/8 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 8/8 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n qemu-kvm.x86_64 10:1.5.3- 2019-12-09T20:29:22.771 INFO:teuthology.task.ansible.out:167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-09T20:29:22.830 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:22 +0000 (0:00:17.244) 0:04:09.614 ******* 2019-12-09T20:29:23.640 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2019-12-09T20:29:23.686 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:23 +0000 (0:00:00.854) 0:04:10.468 ******* 2019-12-09T20:29:24.207 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-12-09T20:29:24.208 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2019-12-09T20:29:24.247 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:24 +0000 (0:00:00.563) 0:04:11.031 ******* 2019-12-09T20:29:24.285 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-09T20:29:24.292 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:24.353 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:24 +0000 (0:00:00.095) 0:04:11.127 ******* 2019-12-09T20:29:24.721 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-09T20:29:24.723 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": ["@core", "@base", "iozone", "qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "msg": "Warning: Group core does not have any packages to install.\nWarning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-114.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-20.el7.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed", "gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "parted-3.1-29.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-61.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is alrea 2019-12-09T20:29:24.725 INFO:teuthology.task.ansible.out:dy installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed", "python-nose-1.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "podman-1.4.4-4.el7.centos.x86_64 providing podman is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-61.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed", "blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-90.el7.centos.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-90.el7.centos.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-90.el7.centos.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-90.el7.centos.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already insta 2019-12-09T20:29:24.726 INFO:teuthology.task.ansible.out:lled", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.65.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed", "python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.ash.fastserv.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.math.princeton.edu\n * updates: mirror.linux.duke.edu\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependenc 2019-12-09T20:29:24.728 INFO:teuthology.task.ansible.out:ies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n librados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 updates 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 updates 435 k\n\nTransaction Summary\n================================================================================\nInstall 3 Packages (+5 Dependent packages)\n\nTotal download size: 8.1 M\nInstalled size: 23 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 23 MB/s | 8.1 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/8 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/8 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/8 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/8 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/8 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/8 \n Installing : iozone-3.424-2_ceph.el7.centos.x86_64 2019-12-09T20:29:24.731 INFO:teuthology.task.ansible.out: 7/8 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 8/8 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/8 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/8 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/8 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/8 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/8 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/8 \n Verifying : boost-random-1.53.0-27.el7.x86_64 7/8 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 8/8 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-09T20:29:24.796 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:24 +0000 (0:00:00.449) 0:04:11.576 ******* 2019-12-09T20:29:24.806 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-12-09T20:29:24.807 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target dbus.service system.slice polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ebtables.service iptables.service ip6tables.service ipset.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Igno 2019-12-09T20:29:24.809 INFO:teuthology.task.ansible.out:reOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", " 2019-12-09T20:29:24.811 INFO:teuthology.task.ansible.out:TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "system.slice network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-09T20:29:24.862 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:24 +0000 (0:00:00.069) 0:04:11.646 ******* 2019-12-09T20:29:25.294 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi081.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-12-09T20:29:25.323 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:25 +0000 (0:00:00.461) 0:04:12.107 ******* 2019-12-09T20:29:25.601 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2019-12-09T20:29:25.648 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:25 +0000 (0:00:00.323) 0:04:12.431 ******* 2019-12-09T20:29:26.130 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-12-09T20:29:26.183 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.536) 0:04:12.967 ******* 2019-12-09T20:29:26.190 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-12-09T20:29:26.190 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2019-12-09T20:29:26.239 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.056) 0:04:13.023 ******* 2019-12-09T20:29:26.248 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-12-09T20:29:26.250 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:26.308 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.068) 0:04:13.091 ******* 2019-12-09T20:29:26.323 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:26.357 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.049) 0:04:13.141 ******* 2019-12-09T20:29:26.363 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-12-09T20:29:26.420 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.051) 0:04:13.192 ******* 2019-12-09T20:29:26.456 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:26.499 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.090) 0:04:13.283 ******* 2019-12-09T20:29:26.541 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-12-09T20:29:26.543 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:26.591 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.092) 0:04:13.375 ******* 2019-12-09T20:29:26.640 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-12-09T20:29:26.642 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:26.685 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.094) 0:04:13.469 ******* 2019-12-09T20:29:26.714 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-12-09T20:29:26.716 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:26.774 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.081) 0:04:13.551 ******* 2019-12-09T20:29:26.817 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-12-09T20:29:26.879 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.111) 0:04:13.663 ******* 2019-12-09T20:29:26.888 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-12-09T20:29:26.890 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target system.slice dbus.service polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "shutdown.target ebtables.service ip6tables.service iptables.service ipset.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Igno 2019-12-09T20:29:26.892 INFO:teuthology.task.ansible.out:reOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", " 2019-12-09T20:29:26.893 INFO:teuthology.task.ansible.out:TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "system.slice network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-09T20:29:26.946 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:26 +0000 (0:00:00.066) 0:04:13.730 ******* 2019-12-09T20:29:26.953 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-12-09T20:29:26.956 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.013 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.067) 0:04:13.797 ******* 2019-12-09T20:29:27.040 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-12-09T20:29:27.041 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.099 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.086) 0:04:13.883 ******* 2019-12-09T20:29:27.126 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-12-09T20:29:27.128 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.176 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.077) 0:04:13.960 ******* 2019-12-09T20:29:27.215 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.264 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.088) 0:04:14.048 ******* 2019-12-09T20:29:27.266 INFO:teuthology.task.ansible.out: 2019-12-09T20:29:27.304 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-12-09T20:29:27.312 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.346 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.078) 0:04:14.127 ******* 2019-12-09T20:29:27.382 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi067.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-12-09T20:29:27.423 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.079) 0:04:14.207 ******* 2019-12-09T20:29:27.460 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.036) 0:04:14.243 ******* 2019-12-09T20:29:27.487 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-12-09T20:29:27.489 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.540 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.081) 0:04:14.324 ******* 2019-12-09T20:29:27.581 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-12-09T20:29:27.582 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.630 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.089) 0:04:14.414 ******* 2019-12-09T20:29:27.670 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-12-09T20:29:27.672 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.710 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.080) 0:04:14.494 ******* 2019-12-09T20:29:27.738 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.804 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.092) 0:04:14.586 ******* 2019-12-09T20:29:27.842 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-12-09T20:29:27.845 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:27.905 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.103) 0:04:14.689 ******* 2019-12-09T20:29:27.914 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-12-09T20:29:27.916 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-12-09T20:29:27.958 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.052) 0:04:14.742 ******* 2019-12-09T20:29:27.967 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.003 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:27 +0000 (0:00:00.044) 0:04:14.787 ******* 2019-12-09T20:29:28.009 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-12-09T20:29:28.012 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.050 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.046) 0:04:14.834 ******* 2019-12-09T20:29:28.057 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-12-09T20:29:28.059 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.100 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.049) 0:04:14.883 ******* 2019-12-09T20:29:28.106 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-12-09T20:29:28.160 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.061) 0:04:14.944 ******* 2019-12-09T20:29:28.169 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-12-09T20:29:28.172 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.210 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.048) 0:04:14.993 ******* 2019-12-09T20:29:28.216 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-12-09T20:29:28.218 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.256 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.047) 0:04:15.041 ******* 2019-12-09T20:29:28.262 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-09T20:29:28.264 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.309 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.047) 0:04:15.088 ******* 2019-12-09T20:29:28.310 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-12-09T20:29:28.317 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.351 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.047) 0:04:15.135 ******* 2019-12-09T20:29:28.356 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-09T20:29:28.358 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.401 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.049) 0:04:15.185 ******* 2019-12-09T20:29:28.406 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-12-09T20:29:28.408 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.450 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.048) 0:04:15.234 ******* 2019-12-09T20:29:28.455 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-09T20:29:28.457 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.494 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.044) 0:04:15.278 ******* 2019-12-09T20:29:28.499 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-12-09T20:29:28.504 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.541 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.044) 0:04:15.323 ******* 2019-12-09T20:29:28.545 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-09T20:29:28.545 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.586 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.046) 0:04:15.370 ******* 2019-12-09T20:29:28.592 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-12-09T20:29:28.636 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.049) 0:04:15.419 ******* 2019-12-09T20:29:28.648 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.683 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.047) 0:04:15.467 ******* 2019-12-09T20:29:28.690 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-12-09T20:29:28.692 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.731 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.047) 0:04:15.515 ******* 2019-12-09T20:29:28.740 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-09T20:29:28.743 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.780 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.048) 0:04:15.564 ******* 2019-12-09T20:29:28.786 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-12-09T20:29:28.787 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.834 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.045) 0:04:15.610 ******* 2019-12-09T20:29:28.838 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.884 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.058) 0:04:15.668 ******* 2019-12-09T20:29:28.895 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.933 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.049) 0:04:15.717 ******* 2019-12-09T20:29:28.940 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-12-09T20:29:28.942 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:28.986 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:28 +0000 (0:00:00.052) 0:04:15.770 ******* 2019-12-09T20:29:28.993 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-09T20:29:28.995 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.034 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.048) 0:04:15.818 ******* 2019-12-09T20:29:29.043 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.090 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.053) 0:04:15.872 ******* 2019-12-09T20:29:29.093 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-12-09T20:29:29.095 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.135 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.047) 0:04:15.919 ******* 2019-12-09T20:29:29.141 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-09T20:29:29.143 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.183 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.047) 0:04:15.966 ******* 2019-12-09T20:29:29.193 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.230 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.048) 0:04:16.014 ******* 2019-12-09T20:29:29.242 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.301 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.069) 0:04:16.084 ******* 2019-12-09T20:29:29.312 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.354 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.054) 0:04:16.138 ******* 2019-12-09T20:29:29.359 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-09T20:29:29.361 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.398 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.044) 0:04:16.182 ******* 2019-12-09T20:29:29.413 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.450 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.052) 0:04:16.235 ******* 2019-12-09T20:29:29.477 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-12-09T20:29:29.479 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.518 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.067) 0:04:16.302 ******* 2019-12-09T20:29:29.545 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-12-09T20:29:29.547 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.591 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.072) 0:04:16.375 ******* 2019-12-09T20:29:29.667 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.692 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.100) 0:04:16.476 ******* 2019-12-09T20:29:29.721 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-12-09T20:29:29.723 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.783 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.090) 0:04:16.567 ******* 2019-12-09T20:29:29.791 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-09T20:29:29.794 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2019-12-09T20:29:29.835 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.051) 0:04:16.619 ******* 2019-12-09T20:29:29.841 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-12-09T20:29:29.844 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.885 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.049) 0:04:16.669 ******* 2019-12-09T20:29:29.925 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:29.995 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:29 +0000 (0:00:00.110) 0:04:16.779 ******* 2019-12-09T20:29:30.022 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-09T20:29:30.023 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.068 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.073) 0:04:16.852 ******* 2019-12-09T20:29:30.097 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.130 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.062) 0:04:16.915 ******* 2019-12-09T20:29:30.164 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.196 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.065) 0:04:16.980 ******* 2019-12-09T20:29:30.202 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-09T20:29:30.203 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-09T20:29:30.248 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.045) 0:04:17.025 ******* TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.294 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.052) 0:04:17.078 ******* 2019-12-09T20:29:30.325 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.357 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.063) 0:04:17.141 ******* 2019-12-09T20:29:30.383 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-09T20:29:30.385 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.420 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.062) 0:04:17.204 ******* 2019-12-09T20:29:30.453 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.482 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.062) 0:04:17.266 ******* 2019-12-09T20:29:30.514 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-12-09T20:29:30.516 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.551 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.069) 0:04:17.336 ******* 2019-12-09T20:29:30.587 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2019-12-09T20:29:30.589 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.624 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.072) 0:04:17.408 ******* 2019-12-09T20:29:30.660 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.035) 0:04:17.444 ******* 2019-12-09T20:29:30.665 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-09T20:29:30.666 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.707 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.046) 0:04:17.491 ******* 2019-12-09T20:29:30.713 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-09T20:29:30.714 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.758 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.051) 0:04:17.542 ******* 2019-12-09T20:29:30.763 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-09T20:29:30.764 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.803 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.044) 0:04:17.587 ******* 2019-12-09T20:29:30.811 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-09T20:29:30.813 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.848 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.045) 0:04:17.632 ******* 2019-12-09T20:29:30.853 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-09T20:29:30.855 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.903 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.055) 0:04:17.687 ******* 2019-12-09T20:29:30.909 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-09T20:29:30.910 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:30.949 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:30 +0000 (0:00:00.045) 0:04:17.733 ******* 2019-12-09T20:29:31.012 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.052 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.102) 0:04:17.836 ******* 2019-12-09T20:29:31.092 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.126 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.074) 0:04:17.910 ******* 2019-12-09T20:29:31.152 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-09T20:29:31.154 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.200 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.072) 0:04:17.982 ******* 2019-12-09T20:29:31.225 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.262 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.064) 0:04:18.046 ******* 2019-12-09T20:29:31.268 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-09T20:29:31.269 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2019-12-09T20:29:31.311 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.047) 0:04:18.094 ******* 2019-12-09T20:29:31.316 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-09T20:29:31.317 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.361 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.048) 0:04:18.142 ******* 2019-12-09T20:29:31.385 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.428 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.066) 0:04:18.209 ******* 2019-12-09T20:29:31.452 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.491 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.066) 0:04:18.275 ******* 2019-12-09T20:29:31.517 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-12-09T20:29:31.520 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.564 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.072) 0:04:18.348 ******* 2019-12-09T20:29:31.596 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-09T20:29:31.598 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.635 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.070) 0:04:18.419 ******* 2019-12-09T20:29:31.650 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-09T20:29:31.651 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-09T20:29:31.695 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.060) 0:04:18.479 ******* 2019-12-09T20:29:31.702 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.740 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.044) 0:04:18.524 ******* 2019-12-09T20:29:31.769 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.805 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.064) 0:04:18.589 ******* 2019-12-09T20:29:31.831 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-12-09T20:29:31.833 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.868 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.062) 0:04:18.652 ******* 2019-12-09T20:29:31.906 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:31.943 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:31 +0000 (0:00:00.074) 0:04:18.727 ******* 2019-12-09T20:29:31.969 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-12-09T20:29:31.971 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.008 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.064) 0:04:18.791 ******* 2019-12-09T20:29:32.034 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-12-09T20:29:32.036 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.071 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.063) 0:04:18.855 ******* 2019-12-09T20:29:32.076 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-09T20:29:32.077 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2019-12-09T20:29:32.122 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.051) 0:04:18.906 ******* 2019-12-09T20:29:32.129 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.173 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.050) 0:04:18.957 ******* 2019-12-09T20:29:32.179 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-09T20:29:32.180 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.222 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.047) 0:04:19.004 ******* 2019-12-09T20:29:32.226 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-12-09T20:29:32.227 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.283 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.062) 0:04:19.067 ******* 2019-12-09T20:29:32.295 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-09T20:29:32.298 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.339 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.056) 0:04:19.123 ******* 2019-12-09T20:29:32.345 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-12-09T20:29:32.346 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.392 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.053) 0:04:19.176 ******* 2019-12-09T20:29:32.400 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-09T20:29:32.401 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.447 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.054) 0:04:19.231 ******* 2019-12-09T20:29:32.453 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-12-09T20:29:32.454 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.497 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.050) 0:04:19.281 ******* 2019-12-09T20:29:32.508 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.561 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.058) 0:04:19.340 ******* 2019-12-09T20:29:32.562 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-09T20:29:32.566 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-12-09T20:29:32.613 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.057) 0:04:19.397 ******* 2019-12-09T20:29:32.619 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-09T20:29:32.624 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.661 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.048) 0:04:19.445 ******* 2019-12-09T20:29:32.670 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-12-09T20:29:32.670 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.672 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-12-09T20:29:32.674 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.689 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.725 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.063) 0:04:19.509 ******* 2019-12-09T20:29:32.731 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-12-09T20:29:32.732 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.736 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.736 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.781 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.056) 0:04:19.565 ******* 2019-12-09T20:29:32.788 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-09T20:29:32.792 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.834 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.052) 0:04:19.618 ******* 2019-12-09T20:29:32.861 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-09T20:29:32.862 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.902 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.068) 0:04:19.687 ******* 2019-12-09T20:29:32.931 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-09T20:29:32.933 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:32.972 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:32 +0000 (0:00:00.069) 0:04:19.756 ******* 2019-12-09T20:29:32.996 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-12-09T20:29:33.002 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003740", "end": "2019-12-09 20:29:32.922812", "rc": 0, "start": "2019-12-09 20:29:32.919072", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-12-09T20:29:33.027 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.054) 0:04:19.811 ******* 2019-12-09T20:29:33.032 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-12-09T20:29:33.034 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.073 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.046) 0:04:19.857 ******* 2019-12-09T20:29:33.079 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-12-09T20:29:33.081 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.127 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.054) 0:04:19.911 ******* 2019-12-09T20:29:33.135 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.175 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.047) 0:04:19.959 ******* 2019-12-09T20:29:33.181 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-12-09T20:29:33.182 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.221 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.045) 0:04:20.005 ******* 2019-12-09T20:29:33.226 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-09T20:29:33.228 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.272 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.051) 0:04:20.057 ******* 2019-12-09T20:29:33.278 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2019-12-09T20:29:33.279 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.317 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.044) 0:04:20.101 ******* 2019-12-09T20:29:33.322 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-09T20:29:33.325 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.381 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.063) 0:04:20.165 ******* 2019-12-09T20:29:33.390 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-12-09T20:29:33.391 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.435 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.051) 0:04:20.216 ******* 2019-12-09T20:29:33.438 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-12-09T20:29:33.441 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.480 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.047) 0:04:20.264 ******* 2019-12-09T20:29:33.518 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.556 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.076) 0:04:20.340 ******* 2019-12-09T20:29:33.604 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-12-09T20:29:33.605 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.650 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.093) 0:04:20.434 ******* 2019-12-09T20:29:33.677 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-12-09T20:29:33.679 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.716 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.066) 0:04:20.500 ******* 2019-12-09T20:29:33.743 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-12-09T20:29:33.744 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.782 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.066) 0:04:20.566 ******* 2019-12-09T20:29:33.810 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.848 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.066) 0:04:20.633 ******* 2019-12-09T20:29:33.883 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:33.927 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.076) 0:04:20.709 ******* 2019-12-09T20:29:33.933 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-12-09T20:29:33.934 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2019-12-09T20:29:33.986 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:33 +0000 (0:00:00.060) 0:04:20.770 ******* 2019-12-09T20:29:33.993 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-12-09T20:29:33.993 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.033 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.047) 0:04:20.817 ******* 2019-12-09T20:29:34.042 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-09T20:29:34.044 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item={u'block_used': 13040430, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227284054, u'size_available': 930955485184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966954, u'device': u'/dev/sda1', u'inode_used': 88022, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227284054, "block_size": 4096, "block_total": 240324484, "block_used": 13040430, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966954, "inode_total": 61054976, "inode_used": 88022, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930955485184, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}} 2019-12-09T20:29:34.100 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.066) 0:04:20.884 ******* 2019-12-09T20:29:34.108 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-12-09T20:29:34.109 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.147 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.047) 0:04:20.931 ******* 2019-12-09T20:29:34.184 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-09T20:29:34.186 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-12-09T20:29:34.222 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.074) 0:04:21.006 ******* 2019-12-09T20:29:34.259 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-12-09T20:29:34.261 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.273 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-12-09T20:29:34.284 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.295 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.305 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.306 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.318 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.359 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.137) 0:04:21.143 ******* 2019-12-09T20:29:34.443 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-12-09T20:29:34.444 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.013905", "end": "2019-12-09 20:29:34.406762", "rc": 0, "start": "2019-12-09 20:29:34.392857", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:34.488 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.128) 0:04:21.272 ******* 2019-12-09T20:29:34.512 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-12-09T20:29:34.513 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004288", "end": "2019-12-09 20:29:34.481497", "rc": 0, "start": "2019-12-09 20:29:34.477209", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-12-09T20:29:34.567 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.077) 0:04:21.349 ******* 2019-12-09T20:29:34.578 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi081.front.sepia.ceph.com] => (item={u'block_used': 13040430, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227284054, u'size_available': 930955485184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966954, u'device': u'/dev/sda1', u'inode_used': 88022, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227284054, "block_size": 4096, "block_total": 240324484, "block_used": 13040430, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966954, "inode_total": 61054976, "inode_used": 88022, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930955485184, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.613 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.047) 0:04:21.396 ******* 2019-12-09T20:29:34.618 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-12-09T20:29:34.620 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.668 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.048) 0:04:21.445 ******* 2019-12-09T20:29:34.670 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-12-09T20:29:34.729 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.067) 0:04:21.513 ******* 2019-12-09T20:29:34.734 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-12-09T20:29:34.736 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.775 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.045) 0:04:21.558 ******* 2019-12-09T20:29:34.783 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-09T20:29:34.784 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5L4BN', u'wwn-0x5000c500918e822f'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5L4BN', u'wwn': u'0x5000c500918e822f', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5L4BN-part1', u'wwn-0x5000c500918e822f-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L4BN", "wwn-0x5000c500918e822f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L4BN-part1", "wwn-0x5000c500918e822f-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L4BN", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-09T20:29:34.786 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c500918e822f"}}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.810 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.843 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.068) 0:04:21.627 ******* 2019-12-09T20:29:34.873 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:34.912 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:34 +0000 (0:00:00.067) 0:04:21.694 ******* 2019-12-09T20:29:35.424 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-12-09T20:29:35.429 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2019-12-09T20:29:35.478 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:35 +0000 (0:00:00.567) 0:04:22.262 ******* 2019-12-09T20:29:35.527 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-09T20:29:35.527 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item={u'block_used': 13040459, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227284025, u'size_available': 930955366400, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966954, u'device': u'/dev/sda1', u'inode_used': 88022, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227284025, "block_size": 4096, "block_total": 240324484, "block_used": 13040459, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966954, "inode_total": 61054976, "inode_used": 88022, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930955366400, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}} 2019-12-09T20:29:35.573 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:35 +0000 (0:00:00.095) 0:04:22.357 ******* 2019-12-09T20:29:35.917 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-12-09T20:29:35.917 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.011756", "end": "2019-12-09 20:29:35.895211", "rc": 0, "start": "2019-12-09 20:29:35.883455", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:35.976 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:35 +0000 (0:00:00.395) 0:04:22.752 ******* 2019-12-09T20:29:36.027 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-12-09T20:29:36.029 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={u'block_used': 13040459, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227284025, u'size_available': 930955366400, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966954, u'device': u'/dev/sda1', u'inode_used': 88022, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227284025, "block_size": 4096, "block_total": 240324484, "block_used": 13040459, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966954, "inode_total": 61054976, "inode_used": 88022, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930955366400, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:36.065 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:36 +0000 (0:00:00.097) 0:04:22.850 ******* 2019-12-09T20:29:36.092 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-12-09T20:29:36.129 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:36 +0000 (0:00:00.063) 0:04:22.913 ******* 2019-12-09T20:29:36.177 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-09T20:29:36.179 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4W31M', u'wwn-0x5000c50087b8c354'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4W31M', u'wwn': u'0x5000c50087b8c354', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4W31M-part1', u'wwn-0x5000c50087b8c354-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W31M", "wwn-0x5000c50087b8c354"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W31M-part1", "wwn-0x5000c50087b8c354-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W31M", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-09T20:29:36.181 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50087b8c354"}}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:36.671 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-EyZ2Ba-0RRy-om4u-fxj5-wNfZ-qQ8S-EIPZ8j', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400N7400BGN', u'nvme-nvme.8086-504846543632303430304e3734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.584482", "end": "2019-12-09 20:29:36.638157", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-EyZ2Ba-0RRy-om4u-fxj5-wNfZ-qQ8S-EIPZ8j", "nvme-INTEL_SSDPEDMD400G4_PHFT620400N7400BGN", "nvme-nvme.8086-504846543632303430304e3734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:35.053 2019-12-09T20:29:36.673 INFO:teuthology.task.ansible.out:675", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:37.932 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-7Rcaxq-XbO6-Ixce-l43U-zcR1-vwUg-6xhV0r', u'nvme-INTEL_SSDPEDMD400G4_CVFT53440042400BGN', u'nvme-nvme.8086-43564654353334343030343234303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.430457", "end": "2019-12-09 20:29:37.885891", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-7Rcaxq-XbO6-Ixce-l43U-zcR1-vwUg-6xhV0r", "nvme-INTEL_SSDPEDMD400G4_CVFT53440042400BGN", "nvme-nvme.8086-43564654353334343030343234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:36.455 2019-12-09T20:29:37.934 INFO:teuthology.task.ansible.out:434", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:38.076 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8GFdw9ksfuEhGzOtxiZVteyqLHfkeOjb1'], u'uuids': [u'21aa835f-1605-4f7f-bf6d-5f3559656b20']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.081173", "end": "2019-12-09 20:29:38.029685", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8GFdw9ksfuEhGzOtxiZVteyqLHfkeOjb1"], "labels": [], "masters": [], "uuids": ["21aa835f-1605-4f7f-bf6d-5f3559656b20"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:36.948512", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:39.301 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAAJe4IJlK0u17qUBulbdyqrn9IC8zTrkg0'], u'uuids': [u'df17a459-c1b8-40aa-baa6-7e2d789113fc']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.044575", "end": "2019-12-09 20:29:39.254978", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAAJe4IJlK0u17qUBulbdyqrn9IC8zTrkg0"], "labels": [], "masters": [], "uuids": ["df17a459-c1b8-40aa-baa6-7e2d789113fc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:38.210403", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:39.457 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8PVJvG35iGuLtHzKuY42dqVDJZjedtQcd'], u'uuids': [u'5837ab7c-bc41-4b18-b2f3-8d2af385eda7']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.088380", "end": "2019-12-09 20:29:39.422733", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8PVJvG35iGuLtHzKuY42dqVDJZjedtQcd"], "labels": [], "masters": [], "uuids": ["5837ab7c-bc41-4b18-b2f3-8d2af385eda7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:38.334353", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:40.696 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAAzLnlY3aUvku5zuUpmFSXPerLrbfOzYw3'], u'uuids': [u'74692d59-025a-44f6-ad8c-94e38fbed1f0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.062083", "end": "2019-12-09 20:29:40.654764", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAAzLnlY3aUvku5zuUpmFSXPerLrbfOzYw3"], "labels": [], "masters": [], "uuids": ["74692d59-025a-44f6-ad8c-94e38fbed1f0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:39.592681", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:40.859 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8hYS6msSICs4v0ZB1CRJFnf6DM8FLGNBK'], u'uuids': [u'a6aafcbd-b0b5-4126-b90f-5b9513b75a87']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.087627", "end": "2019-12-09 20:29:40.824045", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8hYS6msSICs4v0ZB1CRJFnf6DM8FLGNBK"], "labels": [], "masters": [], "uuids": ["a6aafcbd-b0b5-4126-b90f-5b9513b75a87"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:39.736418", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:42.103 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAASLZqqEHwyU2YSq0oQ8V5nzcwGbl5lAvH'], u'uuids': [u'7b679e87-9e0c-4e5e-9cff-53d44cc0aeaa']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.086104", "end": "2019-12-09 20:29:42.063594", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAASLZqqEHwyU2YSq0oQ8V5nzcwGbl5lAvH"], "labels": [], "masters": [], "uuids": ["7b679e87-9e0c-4e5e-9cff-53d44cc0aeaa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:40.977490", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:42.269 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8cX1ViLxxA6y2ulPc2Forx70lr9OJSYw6'], u'uuids': [u'e4e89f96-0e35-4ec6-8504-9a46cfc7926a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.079336", "end": "2019-12-09 20:29:42.224337", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8cX1ViLxxA6y2ulPc2Forx70lr9OJSYw6"], "labels": [], "masters": [], "uuids": ["e4e89f96-0e35-4ec6-8504-9a46cfc7926a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:41.145001", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:43.498 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAA7tO1ZsnVAWf6vLlYvzHYPLd9bMVKooVm'], u'uuids': [u'7a5a7bd5-99ed-412c-ab9e-586e13df22ab']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.048800", "end": "2019-12-09 20:29:43.455879", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAA7tO1ZsnVAWf6vLlYvzHYPLd9bMVKooVm"], "labels": [], "masters": [], "uuids": ["7a5a7bd5-99ed-412c-ab9e-586e13df22ab"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:42.407079", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:43.663 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8h0FLUq2uAEQn0HKjPa63zwINHaOX06yN'], u'uuids': [u'a233c445-b584-4224-9aa7-62b157aba365']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.087004", "end": "2019-12-09 20:29:43.616095", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8h0FLUq2uAEQn0HKjPa63zwINHaOX06yN"], "labels": [], "masters": [], "uuids": ["a233c445-b584-4224-9aa7-62b157aba365"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:42.529091", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:43.709 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:43 +0000 (0:00:07.579) 0:04:30.493 ******* 2019-12-09T20:29:43.754 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-09T20:29:43.755 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5L4BN', u'wwn-0x5000c500918e822f'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5L4BN', u'wwn': u'0x5000c500918e822f', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5L4BN-part1', u'wwn-0x5000c500918e822f-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L4BN", "wwn-0x5000c500918e822f"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5L4BN-part1", "wwn-0x5000c500918e822f-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5L4BN", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-09T20:29:43.757 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c500918e822f"}}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:44.117 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-EyZ2Ba-0RRy-om4u-fxj5-wNfZ-qQ8S-EIPZ8j', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400N7400BGN', u'nvme-nvme.8086-504846543632303430304e3734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.054511", "end": "2019-12-09 20:29:44.084043", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-EyZ2Ba-0RRy-om4u-fxj5-wNfZ-qQ8S-EIPZ8j", "nvme-INTEL_SSDPEDMD400G4_PHFT620400N7400BGN", "nvme-nvme.8086-504846543632303430304e3734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:44.029532", "stderr": 2019-12-09T20:29:44.124 INFO:teuthology.task.ansible.out: "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0483894 s, 2.4 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0483894 s, 2.4 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:44.626 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8GFdw9ksfuEhGzOtxiZVteyqLHfkeOjb1'], u'uuids': [u'21aa835f-1605-4f7f-bf6d-5f3559656b20']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.211541", "end": "2019-12-09 20:29:44.594374", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8GFdw9ksfuEhGzOtxiZVteyqLHfkeOjb1"], "labels": [], "masters": [], "uuids": ["21aa835f-1605-4f7f-bf6d-5f3559656b20"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:44.382833", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.2058 s, 560 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.2058 s, 560 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:44.884 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-09T20:29:44.886 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAArXp8oeu2SEYQ1qxp8AIpNuu65cfwdiq0'], u'uuids': [u'2470a8fa-efdc-449d-9302-fe2cd3a2ba92']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.063980", "end": "2019-12-09 20:29:44.839113", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAArXp8oeu2SEYQ1qxp8AIpNuu65cfwdiq0"], "labels": [], "masters": [], "uuids": ["2470a8fa-efdc-449d-9302-fe2cd3a2ba92"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:43.775133", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-09T20:29:44.951 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:44 +0000 (0:00:01.242) 0:04:31.735 ******* 2019-12-09T20:29:45.004 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-09T20:29:45.006 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4W31M', u'wwn-0x5000c50087b8c354'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4W31M', u'wwn': u'0x5000c50087b8c354', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4W31M-part1', u'wwn-0x5000c50087b8c354-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W31M", "wwn-0x5000c50087b8c354"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W31M-part1", "wwn-0x5000c50087b8c354-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W31M", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-09T20:29:45.007 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50087b8c354"}}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:45.146 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8PVJvG35iGuLtHzKuY42dqVDJZjedtQcd'], u'uuids': [u'5837ab7c-bc41-4b18-b2f3-8d2af385eda7']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.201167", "end": "2019-12-09 20:29:45.106091", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8PVJvG35iGuLtHzKuY42dqVDJZjedtQcd"], "labels": [], "masters": [], "uuids 2019-12-09T20:29:45.147 INFO:teuthology.task.ansible.out:": ["5837ab7c-bc41-4b18-b2f3-8d2af385eda7"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:44.904924", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.194682 s, 592 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.194682 s, 592 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:45.447 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-7Rcaxq-XbO6-Ixce-l43U-zcR1-vwUg-6xhV0r', u'nvme-INTEL_SSDPEDMD400G4_CVFT53440042400BGN', u'nvme-nvme.8086-43564654353334343030343234303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.084485", "end": "2019-12-09 20:29:45.376439", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-7Rcaxq-XbO6-Ixce-l43U-zcR1-vwUg-6xhV0r", "nvme-INTEL_SSDPEDMD400G4_CVFT53440042400BGN", "nvme-nvme.8086-43564654353334343030343234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:45.291954", "stderr": 2019-12-09T20:29:45.450 INFO:teuthology.task.ansible.out: "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0777847 s, 1.5 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0777847 s, 1.5 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:45.682 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8hYS6msSICs4v0ZB1CRJFnf6DM8FLGNBK'], u'uuids': [u'a6aafcbd-b0b5-4126-b90f-5b9513b75a87']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.194955", "end": "2019-12-09 20:29:45.637751", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8hYS6msSICs4v0ZB1CRJFnf6DM8FLGNBK"], "labels": [], "masters": [], "uuids": ["a6aafcbd-b0b5-4126-b90f-5b9513b75a87"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:45.442796", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.18859 s, 612 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.18859 s, 612 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:45.947 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAAJe4IJlK0u17qUBulbdyqrn9IC8zTrkg0'], u'uuids': [u'df17a459-c1b8-40aa-baa6-7e2d789113fc']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.217776", "end": "2019-12-09 20:29:45.908894", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAAJe4IJlK0u17qUBulbdyqrn9IC8zTrkg0"], "labels": [], "masters": [], "uuids": ["df17a459-c1b8-40aa-baa6-7e2d789113fc"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:45.691118", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.211949 s, 544 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.211949 s, 544 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:46.148 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8cX1ViLxxA6y2ulPc2Forx70lr9OJSYw6'], u'uuids': [u'e4e89f96-0e35-4ec6-8504-9a46cfc7926a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.172922", "end": "2019-12-09 20:29:46.118085", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8cX1ViLxxA6y2ulPc2Forx70lr9OJSYw6"], "labels": [], "masters": [], "uuids": ["e4e89f96-0e35-4ec6-8504-9a46cfc7926a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:45.945163", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.167228 s, 690 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.167228 s, 690 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:46.510 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAAzLnlY3aUvku5zuUpmFSXPerLrbfOzYw3'], u'uuids': [u'74692d59-025a-44f6-ad8c-94e38fbed1f0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.220403", "end": "2019-12-09 20:29:46.463724", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAAzLnlY3aUvku5zuUpmFSXPerLrbfOzYw3"], "labels": [], "masters": [], "uuids": ["74692d59-025a-44f6-ad8c-94e38fbed1f0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:46.243321", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.21369 s, 540 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.21369 s, 540 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:46.660 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8h0FLUq2uAEQn0HKjPa63zwINHaOX06yN'], u'uuids': [u'a233c445-b584-4224-9aa7-62b157aba365']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400N7400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.180176", "end": "2019-12-09 20:29:46.622539", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-0XXtB8vR8YUEsQG3l3lPGA1ycB1JDnf8h0FLUq2uAEQn0HKjPa63zwINHaOX06yN"], "labels": [], "masters": [], "uuids": ["a233c445-b584-4224-9aa7-62b157aba365"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400N7400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:46.442363", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.174385 s, 661 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.174385 s, 661 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:46.709 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:46 +0000 (0:00:01.757) 0:04:33.493 ******* 2019-12-09T20:29:47.009 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAASLZqqEHwyU2YSq0oQ8V5nzcwGbl5lAvH'], u'uuids': [u'7b679e87-9e0c-4e5e-9cff-53d44cc0aeaa']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.193995", "end": "2019-12-09 20:29:46.974107", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAASLZqqEHwyU2YSq0oQ8V5nzcwGbl5lAvH"], "labels": [], "masters": [], "uuids": ["7b679e87-9e0c-4e5e-9cff-53d44cc0aeaa"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:46.780112", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.187471 s, 615 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.187471 s, 615 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:47.125 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-12-09T20:29:47.127 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.086239", "end": "2019-12-09 20:29:47.082730", "rc": 0, "start": "2019-12-09 20:29:46.996491", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:47.189 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:47 +0000 (0:00:00.480) 0:04:33.973 ******* 2019-12-09T20:29:47.191 INFO:teuthology.task.ansible.out: 2019-12-09T20:29:47.480 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-09T20:29:47.481 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAA7tO1ZsnVAWf6vLlYvzHYPLd9bMVKooVm'], u'uuids': [u'7a5a7bd5-99ed-412c-ab9e-586e13df22ab']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.179282", "end": "2019-12-09 20:29:47.447838", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAA7tO1ZsnVAWf6vLlYvzHYPLd9bMVKooVm"], "labels": [], "masters": [], "uuids": ["7a5a7bd5-99ed-412c-ab9e-586e13df22ab"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:47.268556", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.174356 s, 662 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.174356 s, 662 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:47.572 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-12-09T20:29:47.574 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.054470", "end": "2019-12-09 20:29:47.538791", "rc": 0, "start": "2019-12-09 20:29:47.484321", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:47.626 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:47 +0000 (0:00:00.432) 0:04:34.406 ******* 2019-12-09T20:29:47.651 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-12-09T20:29:47.702 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:47 +0000 (0:00:00.079) 0:04:34.486 ******* 2019-12-09T20:29:47.730 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:47.770 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:47 +0000 (0:00:00.068) 0:04:34.554 ******* 2019-12-09T20:29:47.797 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:47.837 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:47 +0000 (0:00:00.066) 0:04:34.621 ******* 2019-12-09T20:29:47.867 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:47.916 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:47 +0000 (0:00:00.078) 0:04:34.699 ******* 2019-12-09T20:29:47.960 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:48.009 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.093) 0:04:34.793 ******* 2019-12-09T20:29:48.026 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-09T20:29:48.027 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAArXp8oeu2SEYQ1qxp8AIpNuu65cfwdiq0'], u'uuids': [u'2470a8fa-efdc-449d-9302-fe2cd3a2ba92']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'CVFT53440042400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.205712", "end": "2019-12-09 20:29:47.982317", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-nIx03vP9MKdLfPQg7Tg9b2bVVFhqxgAArXp8oeu2SEYQ1qxp8AIpNuu65cfwdiq0"], "labels": [], "masters": [], "uuids": ["2470a8fa-efdc-449d-9302-fe2cd3a2ba92"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "CVFT53440042400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-09 20:29:47.776605", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.19893 s, 580 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.19893 s, 580 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:48.081 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.069) 0:04:34.862 ******* 2019-12-09T20:29:48.326 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi081.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-09T20:29:48.368 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.289) 0:04:35.152 ******* 2019-12-09T20:29:48.405 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-09T20:29:48.408 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={u'block_used': 13040430, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227284054, u'size_available': 930955485184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966954, u'device': u'/dev/sda1', u'inode_used': 88022, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227284054, "block_size": 4096, "block_total": 240324484, "block_used": 13040430, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966954, "inode_total": 61054976, "inode_used": 88022, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930955485184, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:48.448 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.080) 0:04:35.232 ******* 2019-12-09T20:29:48.449 INFO:teuthology.task.ansible.out: 2019-12-09T20:29:48.477 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-12-09T20:29:48.479 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:48.516 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.067) 0:04:35.300 ******* 2019-12-09T20:29:48.523 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-12-09T20:29:48.524 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.087397", "end": "2019-12-09 20:29:48.456473", "rc": 0, "start": "2019-12-09 20:29:48.369076", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:48.565 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.048) 0:04:35.349 ******* 2019-12-09T20:29:48.571 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-09T20:29:48.573 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:48.612 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.047) 0:04:35.396 ******* 2019-12-09T20:29:48.649 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-09T20:29:48.651 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:48.687 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.074) 0:04:35.471 ******* 2019-12-09T20:29:48.725 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-12-09T20:29:48.727 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:48.764 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.077) 0:04:35.548 ******* 2019-12-09T20:29:48.790 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-12-09T20:29:48.792 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:48.829 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.065) 0:04:35.613 ******* 2019-12-09T20:29:48.866 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-12-09T20:29:48.868 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:48.915 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.086) 0:04:35.699 ******* 2019-12-09T20:29:48.956 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.059632", "end": "2019-12-09 20:29:48.924110", "rc": 0, "start": "2019-12-09 20:29:48.864478", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:29:48.991 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:48 +0000 (0:00:00.076) 0:04:35.776 ******* 2019-12-09T20:29:49.019 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-12-09T20:29:49.061 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:49 +0000 (0:00:00.069) 0:04:35.845 ******* 2019-12-09T20:29:49.101 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-12-09T20:29:49.102 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:49.147 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:49 +0000 (0:00:00.086) 0:04:35.931 ******* 2019-12-09T20:29:49.175 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-12-09T20:29:49.176 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:49.218 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:49 +0000 (0:00:00.070) 0:04:36.002 ******* 2019-12-09T20:29:49.246 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-09T20:29:49.249 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:49.309 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:49 +0000 (0:00:00.090) 0:04:36.092 ******* 2019-12-09T20:29:49.347 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-12-09T20:29:49.349 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:49.384 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:49 +0000 (0:00:00.076) 0:04:36.168 ******* 2019-12-09T20:29:49.750 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2019-12-09T20:29:49.752 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-09T20:29:49.790 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:49 +0000 (0:00:00.405) 0:04:36.574 ******* 2019-12-09T20:29:49.797 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-12-09T20:29:49.798 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2019-12-09T20:29:49.845 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:49 +0000 (0:00:00.052) 0:04:36.627 ******* 2019-12-09T20:29:49.852 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi067.front.sepia.ceph.com] => (item={u'block_used': 13040459, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227284025, u'size_available': 930955366400, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966954, u'device': u'/dev/sda1', u'inode_used': 88022, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227284025, "block_size": 4096, "block_total": 240324484, "block_used": 13040459, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966954, "inode_total": 61054976, "inode_used": 88022, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930955366400, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:49.896 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:49 +0000 (0:00:00.052) 0:04:36.680 ******* 2019-12-09T20:29:49.932 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-12-09T20:29:49.934 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:49.973 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:49 +0000 (0:00:00.076) 0:04:36.756 ******* 2019-12-09T20:29:50.000 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-09T20:29:50.002 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:50.042 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:50 +0000 (0:00:00.070) 0:04:36.827 ******* 2019-12-09T20:29:50.098 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:50.157 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:50 +0000 (0:00:00.112) 0:04:36.939 ******* 2019-12-09T20:29:50.209 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-12-09T20:29:50.211 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:50.262 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:50 +0000 (0:00:00.106) 0:04:37.046 ******* 2019-12-09T20:29:50.300 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-12-09T20:29:50.302 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:50.344 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:50 +0000 (0:00:00.082) 0:04:37.129 ******* 2019-12-09T20:29:50.384 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-12-09T20:29:50.385 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:50.430 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:50 +0000 (0:00:00.085) 0:04:37.214 ******* 2019-12-09T20:29:50.715 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:51.014 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2019-12-09T20:29:51.060 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:51 +0000 (0:00:00.628) 0:04:37.843 ******* 2019-12-09T20:29:51.238 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:51.570 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:51.695 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:52.087 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:52.182 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:52.562 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:52.652 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:52.703 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:52 +0000 (0:00:01.643) 0:04:39.487 ******* 2019-12-09T20:29:53.010 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:53.022 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-09T20:29:53.024 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-12-09T20:29:53.062 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:53 +0000 (0:00:00.359) 0:04:39.846 ******* 2019-12-09T20:29:53.102 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-09T20:29:53.104 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:53.421 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-09T20:29:53.520 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-09T20:29:53.579 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:53 +0000 (0:00:00.516) 0:04:40.363 ******* 2019-12-09T20:29:53.740 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-09T20:29:53.741 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-09T20:29:53.922 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-09T20:29:53.924 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-12-09T20:29:53.962 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:53 +0000 (0:00:00.381) 0:04:40.745 ******* 2019-12-09T20:29:54.021 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-09T20:29:54.023 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:54.043 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-09T20:29:54.327 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-09T20:29:54.409 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-09T20:29:54.466 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:54 +0000 (0:00:00.505) 0:04:41.250 ******* 2019-12-09T20:29:54.623 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-09T20:29:54.793 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-12-09T20:29:54.794 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-09T20:29:54.837 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:54 +0000 (0:00:00.371) 0:04:41.622 ******* 2019-12-09T20:29:54.885 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-12-09T20:29:54.887 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-12-09T20:29:54.949 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:54 +0000 (0:00:00.111) 0:04:41.733 ******* 2019-12-09T20:29:54.956 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-09T20:29:54.958 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-09T20:29:55.286 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-09T20:29:55.328 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:55 +0000 (0:00:00.378) 0:04:42.112 ******* 2019-12-09T20:29:55.664 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-09T20:29:55.708 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:55 +0000 (0:00:00.379) 0:04:42.492 ******* 2019-12-09T20:29:55.718 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi081.front.sepia.ceph.com] => {"changed": true} 2019-12-09T20:29:55.770 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:55 +0000 (0:00:00.061) 0:04:42.553 ******* 2019-12-09T20:29:55.776 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-12-09T20:29:55.778 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-12-09T20:29:55.820 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:55 +0000 (0:00:00.050) 0:04:42.604 ******* 2019-12-09T20:29:56.271 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-12-09T20:29:56.272 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true} 2019-12-09T20:29:56.319 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:56 +0000 (0:00:00.498) 0:04:43.103 ******* 2019-12-09T20:29:56.525 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-12-09T20:29:56.527 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-12-09T20:29:56.563 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:56 +0000 (0:00:00.244) 0:04:43.347 ******* 2019-12-09T20:29:56.600 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-12-09T20:29:56.602 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:56.640 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:56 +0000 (0:00:00.077) 0:04:43.424 ******* 2019-12-09T20:29:56.681 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:56.725 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:56 +0000 (0:00:00.084) 0:04:43.509 ******* 2019-12-09T20:29:56.756 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:56.801 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:56 +0000 (0:00:00.076) 0:04:43.585 ******* 2019-12-09T20:29:56.837 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:56 +0000 (0:00:00.035) 0:04:43.621 ******* 2019-12-09T20:29:56.845 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:56.882 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:56 +0000 (0:00:00.045) 0:04:43.666 ******* 2019-12-09T20:29:56.888 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-12-09T20:29:56.890 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:56.933 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:56 +0000 (0:00:00.050) 0:04:43.717 ******* 2019-12-09T20:29:56.940 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-12-09T20:29:56.942 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:56.980 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:56 +0000 (0:00:00.046) 0:04:43.764 ******* 2019-12-09T20:29:56.989 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.028 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.047) 0:04:43.812 ******* 2019-12-09T20:29:57.036 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-12-09T20:29:57.037 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.079 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.050) 0:04:43.862 ******* 2019-12-09T20:29:57.088 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.126 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.047) 0:04:43.910 ******* 2019-12-09T20:29:57.132 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-12-09T20:29:57.133 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.172 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.045) 0:04:43.956 ******* 2019-12-09T20:29:57.180 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.220 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.044) 0:04:44.000 ******* 2019-12-09T20:29:57.222 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-12-09T20:29:57.223 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.268 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.052) 0:04:44.053 ******* 2019-12-09T20:29:57.297 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-12-09T20:29:57.299 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.340 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.071) 0:04:44.124 ******* 2019-12-09T20:29:57.380 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-12-09T20:29:57.381 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.421 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.080) 0:04:44.205 ******* 2019-12-09T20:29:57.710 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-12-09T20:29:57.718 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2019-12-09T20:29:57.760 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.339) 0:04:44.544 ******* 2019-12-09T20:29:57.791 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-12-09T20:29:57.793 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.847 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.086) 0:04:44.631 ******* 2019-12-09T20:29:57.885 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-12-09T20:29:57.888 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:57.930 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.082) 0:04:44.714 ******* 2019-12-09T20:29:57.935 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-12-09T20:29:57.936 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2019-12-09T20:29:57.980 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:57 +0000 (0:00:00.050) 0:04:44.764 ******* 2019-12-09T20:29:58.019 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-12-09T20:29:58.021 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:58.062 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:58 +0000 (0:00:00.081) 0:04:44.846 ******* 2019-12-09T20:29:58.116 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:58.163 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:58 +0000 (0:00:00.100) 0:04:44.946 ******* 2019-12-09T20:29:58.951 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2019-12-09T20:29:59.016 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:59 +0000 (0:00:00.852) 0:04:45.799 ******* 2019-12-09T20:29:59.047 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-12-09T20:29:59.048 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:59.098 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:59 +0000 (0:00:00.082) 0:04:45.882 ******* 2019-12-09T20:29:59.138 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2019-12-09T20:29:59.149 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2019-12-09T20:29:59.179 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:59 +0000 (0:00:00.080) 0:04:45.963 ******* 2019-12-09T20:29:59.206 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:29:59.246 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:59 +0000 (0:00:00.066) 0:04:46.030 ******* 2019-12-09T20:29:59.607 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-12-09T20:29:59.610 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice sntp.service -.mount syslog.target systemd-journald.socket basic.target tmp.mount ntpdate.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target multi-user.target chronyd.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2019-12-09T20:29:59.612 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2019-12-09T20:29:59.614 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-09T20:29:59.663 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:59 +0000 (0:00:00.417) 0:04:46.447 ******* 2019-12-09T20:29:59.774 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "tmp.mount system.slice basic.target syslog.target systemd-journald.socket ntpdate.service sntp.service -.mount", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "chronyd.service multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2019-12-09T20:29:59.775 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2019-12-09T20:29:59.777 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-09T20:29:59.815 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:29:59 +0000 (0:00:00.152) 0:04:46.599 ******* 2019-12-09T20:30:00.540 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2019-12-09T20:30:00.580 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:00 +0000 (0:00:00.764) 0:04:47.364 ******* 2019-12-09T20:30:00.624 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-12-09T20:30:00.625 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:00.664 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:00 +0000 (0:00:00.084) 0:04:47.448 ******* 2019-12-09T20:30:00.690 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2019-12-09T20:30:00.692 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:00.730 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:00 +0000 (0:00:00.066) 0:04:47.514 ******* 2019-12-09T20:30:00.735 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-12-09T20:30:00.736 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2019-12-09T20:30:00.804 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:00 +0000 (0:00:00.056) 0:04:47.571 ******* 2019-12-09T20:30:00.836 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-12-09T20:30:00.839 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:00.876 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:00 +0000 (0:00:00.089) 0:04:47.661 ******* 2019-12-09T20:30:00.913 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2019-12-09T20:30:00.915 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:00.970 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:00 +0000 (0:00:00.093) 0:04:47.755 ******* 2019-12-09T20:30:01.063 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-12-09T20:30:01.065 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-12-09T20:30:01.106 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:01 +0000 (0:00:00.135) 0:04:47.890 ******* 2019-12-09T20:30:01.309 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-12-09T20:30:01.361 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:01 +0000 (0:00:00.254) 0:04:48.144 ******* 2019-12-09T20:30:01.997 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-12-09T20:30:01.998 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-12-09T20:30:02.074 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:02 +0000 (0:00:00.711) 0:04:48.856 ******* 2019-12-09T20:30:02.310 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-12-09T20:30:02.360 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:02 +0000 (0:00:00.284) 0:04:49.140 ******* 2019-12-09T20:30:03.923 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.3.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.3.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-12-09T20:30:03.999 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:03 +0000 (0:00:01.640) 0:04:50.781 ******* 2019-12-09T20:30:04.008 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.3.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.3.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-12-09T20:30:04.055 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.058) 0:04:50.840 ******* 2019-12-09T20:30:04.065 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-12-09T20:30:04.066 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:04.103 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.047) 0:04:50.887 ******* 2019-12-09T20:30:04.108 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:04.153 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.050) 0:04:50.937 ******* 2019-12-09T20:30:04.163 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-12-09T20:30:04.164 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-12-09T20:30:04.206 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.052) 0:04:50.990 ******* 2019-12-09T20:30:04.212 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-12-09T20:30:04.258 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.051) 0:04:51.042 ******* 2019-12-09T20:30:04.502 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi081.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-12-09T20:30:04.553 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.295) 0:04:51.337 ******* 2019-12-09T20:30:04.592 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-12-09T20:30:04.594 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:04.650 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.096) 0:04:51.434 ******* 2019-12-09T20:30:04.656 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-12-09T20:30:04.656 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-12-09T20:30:04.705 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.052) 0:04:51.487 ******* 2019-12-09T20:30:04.715 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-12-09T20:30:04.716 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-12-09T20:30:04.758 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.055) 0:04:51.542 ******* 2019-12-09T20:30:04.764 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-12-09T20:30:04.766 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:04.806 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.047) 0:04:51.590 ******* 2019-12-09T20:30:04.814 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-12-09T20:30:04.816 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:04.857 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.051) 0:04:51.641 ******* 2019-12-09T20:30:04.863 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-12-09T20:30:04.865 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-12-09T20:30:04.905 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.048) 0:04:51.689 ******* 2019-12-09T20:30:04.932 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-12-09T20:30:04.934 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:04.983 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:04 +0000 (0:00:00.077) 0:04:51.767 ******* 2019-12-09T20:30:05.167 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-12-09T20:30:05.169 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-09T20:30:05.206 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:05 +0000 (0:00:00.222) 0:04:51.990 ******* 2019-12-09T20:30:05.316 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-09T20:30:05.354 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:05 +0000 (0:00:00.147) 0:04:52.138 ******* 2019-12-09T20:30:05.526 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-12-09T20:30:05.527 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-12-09T20:30:05.568 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:05 +0000 (0:00:00.213) 0:04:52.352 ******* 2019-12-09T20:30:05.595 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-12-09T20:30:05.596 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.650 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:05 +0000 (0:00:00.082) 0:04:52.434 ******* 2019-12-09T20:30:05.695 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi081.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.702 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-12-09T20:30:05.703 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi081.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.705 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.746 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:05 +0000 (0:00:00.095) 0:04:52.530 ******* 2019-12-09T20:30:05.758 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.816 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:05 +0000 (0:00:00.069) 0:04:52.600 ******* 2019-12-09T20:30:05.823 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-12-09T20:30:05.825 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.890 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:05 +0000 (0:00:00.074) 0:04:52.674 ******* 2019-12-09T20:30:05.898 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-12-09T20:30:05.899 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.904 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.914 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi081.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.964 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:05 +0000 (0:00:00.073) 0:04:52.748 ******* 2019-12-09T20:30:05.969 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-12-09T20:30:05.970 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.971 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-12-09T20:30:05.972 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi067.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.011 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.047) 0:04:52.795 ******* 2019-12-09T20:30:06.018 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-09T20:30:06.018 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.065 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.053) 0:04:52.849 ******* 2019-12-09T20:30:06.072 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-12-09T20:30:06.074 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.075 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.077 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.078 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.090 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.092 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-12-09T20:30:06.093 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.131 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.066) 0:04:52.915 ******* 2019-12-09T20:30:06.175 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.043) 0:04:52.959 ******* 2019-12-09T20:30:06.181 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-09T20:30:06.181 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.222 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.047) 0:04:53.006 ******* 2019-12-09T20:30:06.229 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-12-09T20:30:06.229 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.273 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.050) 0:04:53.057 ******* 2019-12-09T20:30:06.282 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.346 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.072) 0:04:53.130 ******* 2019-12-09T20:30:06.353 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.405 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.059) 0:04:53.189 ******* 2019-12-09T20:30:06.410 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-12-09T20:30:06.412 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.456 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.051) 0:04:53.240 ******* 2019-12-09T20:30:06.466 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-12-09T20:30:06.467 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.532 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.075) 0:04:53.316 ******* 2019-12-09T20:30:06.539 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.576 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.044) 0:04:53.360 ******* 2019-12-09T20:30:06.582 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-09T20:30:06.584 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.627 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.051) 0:04:53.412 ******* 2019-12-09T20:30:06.637 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.682 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.054) 0:04:53.466 ******* 2019-12-09T20:30:06.690 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.741 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.058) 0:04:53.525 ******* 2019-12-09T20:30:06.747 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-09T20:30:06.749 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.791 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.049) 0:04:53.575 ******* 2019-12-09T20:30:06.797 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-09T20:30:06.798 INFO:teuthology.task.ansible.out:skipping: [smithi081.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.839 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.047) 0:04:53.623 ******* 2019-12-09T20:30:06.847 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-12-09T20:30:06.848 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.900 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.061) 0:04:53.684 ******* 2019-12-09T20:30:06.929 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-09T20:30:06.929 INFO:teuthology.task.ansible.out:skipping: [smithi067.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-09T20:30:06.976 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:06 +0000 (0:00:00.072) 0:04:53.757 ******* 2019-12-09T20:30:07.189 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi081.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.014562", "end": "2019-12-09 20:30:07.143248", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-12-09 20:30:07.128686", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:30:07.494 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.007083", "end": "2019-12-09 20:30:07.455027", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-12-09 20:30:07.447944", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:30:07.792 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.006892", "end": "2019-12-09 20:30:07.753674", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-12-09 20:30:07.746782", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-12-09T20:30:08.099 INFO:teuthology.task.ansible.out:ok: [smithi081.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.007405", "end": "2019-12-09 20:30:08.060100", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-12-09 20:30:08.052695", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:30:08.164 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:08 +0000 (0:00:01.191) 0:04:54.949 ******* 2019-12-09T20:30:08.194 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-12-09T20:30:08.230 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:08 +0000 (0:00:00.065) 0:04:55.014 ******* 2019-12-09T20:30:08.351 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-12-09T20:30:08.373 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:01.017399", "end": "2019-12-09 20:30:08.310968", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-12-09 20:30:07.293569", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:30:08.551 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-12-09T20:30:08.553 INFO:teuthology.task.ansible.out:changed: [smithi081.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-09T20:30:09.677 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-12-09T20:30:09.679 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:01.008355", "end": "2019-12-09 20:30:09.636690", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-12-09 20:30:08.628335", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:30:09.998 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.006278", "end": "2019-12-09 20:30:09.957439", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-12-09 20:30:09.951161", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-12-09T20:30:10.308 INFO:teuthology.task.ansible.out:ok: [smithi067.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.009244", "end": "2019-12-09 20:30:10.274210", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-12-09 20:30:10.264966", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-09T20:30:10.356 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:10 +0000 (0:00:02.126) 0:04:57.140 ******* 2019-12-09T20:30:10.383 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-12-09T20:30:10.422 INFO:teuthology.task.ansible.out:Monday 09 December 2019 20:30:10 +0000 (0:00:00.065) 0:04:57.206 ******* 2019-12-09T20:30:10.740 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-12-09T20:30:10.741 INFO:teuthology.task.ansible.out:changed: [smithi067.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-09T20:30:10.848 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2019-12-09T20:30:10.850 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** 2019-12-09T20:30:10.852 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2019-12-09T20:30:10.853 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi067.front.sepia.ceph.com : ok=121 changed=41 unreachable=0 failed=0 skipped=160 rescued=0 ignored=3 smithi081.front.sepia.ceph.com : ok=121 changed=41 unreachable=0 failed=0 skipped=160 rescued=0 ignored=3 Monday 09 December 2019 20:30:10 +0000 (0:00:00.430) 0:04:57.637 ******* =============================================================================== 2019-12-09T20:30:10.855 INFO:teuthology.task.ansible.out:users : Create all admin users with sudo access. ----------------------- 36.57s users : Update authorized_keys using the keys repo --------------------- 35.46s common : nrpe - Ensure SELinux policy is up to date -------------------- 27.28s 2019-12-09T20:30:10.856 INFO:teuthology.task.ansible.out:testnode : Install packages -------------------------------------------- 17.24s users : Remove revoked users ------------------------------------------- 16.62s common : Update yum cache ---------------------------------------------- 12.42s common : nrpe - Remove SELinux policy package --------------------------- 7.88s testnode : Zap all non-root disks --------------------------------------- 7.58s common : nrpe - Load SELinux policy package ----------------------------- 7.31s testnode : Ensure ceph packages are not present. ------------------------ 7.18s common : Make sure smartmontools is installed --------------------------- 3.53s Gathering Facts --------------------------------------------------------- 3.42s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 3.17s ansible-managed : Add authorized keys for the ansible user. ------------- 3.03s testnode : Ensure ceph dependency packages are not present. ------------- 2.39s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 2.35s common : Configure local epel mirrorlists ------------------------------- 2.35s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 2.33s users : Update authorized_keys for each user with literal keys ---------- 2.31s common : Configure epel repos in /etc/yum.repos.d/ ---------------------- 2.03s 2019-12-09T20:30:11.138 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi081.front.sepia.ceph.com'), Remote(name='ubuntu@smithi067.front.sepia.ceph.com')] 2019-12-09T20:30:11.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:30:11.142 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi081.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:30:11.239 INFO:teuthology.orchestra.run.smithi081:> true 2019-12-09T20:30:11.621 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-09T20:30:11.622 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi067.front.sepia.ceph.com', 'timeout': 60} 2019-12-09T20:30:11.731 INFO:teuthology.orchestra.run.smithi067:> true 2019-12-09T20:30:12.098 INFO:teuthology.run_tasks:Running task clock... 2019-12-09T20:30:12.106 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-12-09T20:30:12.106 INFO:teuthology.orchestra.run.smithi081:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-12-09T20:30:12.242 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-12-09T20:30:22.326 INFO:teuthology.orchestra.run.smithi081.stdout:ntpd: time slew -0.000255s 2019-12-09T20:30:22.365 INFO:teuthology.orchestra.run.smithi081.stderr:Failed to start ntp.service: Unit not found. 2019-12-09T20:30:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: remote refid st t when poll reach delay offset jitter 2019-12-09T20:30:22.440 INFO:teuthology.orchestra.run.smithi081.stdout:============================================================================== 2019-12-09T20:30:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-09T20:30:22.440 INFO:teuthology.orchestra.run.smithi081.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-09T20:30:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-09T20:30:22.441 INFO:teuthology.orchestra.run.smithi081.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-09T20:30:22.442 INFO:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-12-09T20:30:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-12-09T20:30:32.677 INFO:teuthology.orchestra.run.smithi067.stdout:ntpd: time slew +0.002306s 2019-12-09T20:30:32.715 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit not found. 2019-12-09T20:30:32.798 INFO:teuthology.orchestra.run.smithi067.stdout: remote refid st t when poll reach delay offset jitter 2019-12-09T20:30:32.799 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================== 2019-12-09T20:30:32.799 INFO:teuthology.orchestra.run.smithi067.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-09T20:30:32.799 INFO:teuthology.orchestra.run.smithi067.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-09T20:30:32.799 INFO:teuthology.orchestra.run.smithi067.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-09T20:30:32.799 INFO:teuthology.orchestra.run.smithi067.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-09T20:30:32.801 INFO:teuthology.run_tasks:Running task install... 2019-12-09T20:30:32.813 DEBUG:teuthology.task.install:project ceph 2019-12-09T20:30:32.813 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'notcmalloc', 'sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b'}} 2019-12-09T20:30:32.813 DEBUG:teuthology.task.install:config {'flavor': 'notcmalloc', 'sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b'} 2019-12-09T20:30:32.814 INFO:teuthology.task.install:Using flavor: notcmalloc 2019-12-09T20:30:32.822 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-daemon', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-daemon', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2019-12-09T20:30:32.822 INFO:teuthology.task.install:extra packages: [] 2019-12-09T20:30:32.822 INFO:teuthology.orchestra.run.smithi081:> sudo lsb_release -is 2019-12-09T20:30:32.990 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS 2019-12-09T20:30:32.992 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-09T20:30:32.992 INFO:teuthology.orchestra.run.smithi067:> sudo lsb_release -is 2019-12-09T20:30:32.992 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'notcmalloc', 'extras': None} 2019-12-09T20:30:32.993 INFO:teuthology.orchestra.run.smithi081:> sudo lsb_release -is 2019-12-09T20:30:33.130 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS 2019-12-09T20:30:33.130 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-09T20:30:33.131 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=827e03c8a6b1e17d93dcaf79724437bac291c35b 2019-12-09T20:30:33.172 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS 2019-12-09T20:30:33.172 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-09T20:30:33.172 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'notcmalloc', 'extras': None} 2019-12-09T20:30:33.173 INFO:teuthology.orchestra.run.smithi067:> sudo lsb_release -is 2019-12-09T20:30:33.298 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-abhi-testing-2019-12-04-1433/827e03c8a6b1e17d93dcaf79724437bac291c35b/centos/7/flavors/notcmalloc/ 2019-12-09T20:30:33.298 INFO:teuthology.task.install.rpm:Package version is 15.0.0-7936.g827e03c 2019-12-09T20:30:33.312 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS 2019-12-09T20:30:33.313 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-09T20:30:33.314 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=827e03c8a6b1e17d93dcaf79724437bac291c35b 2019-12-09T20:30:33.423 INFO:teuthology.orchestra.run.smithi081:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-12-09T20:30:33.487 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-abhi-testing-2019-12-04-1433/827e03c8a6b1e17d93dcaf79724437bac291c35b/centos/7/flavors/notcmalloc/ 2019-12-09T20:30:33.487 INFO:teuthology.task.install.rpm:Package version is 15.0.0-7936.g827e03c 2019-12-09T20:30:33.560 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-daemon, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse on remote rpm x86_64 2019-12-09T20:30:33.560 INFO:teuthology.orchestra.run.smithi081:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/827e03c8a6b1e17d93dcaf79724437bac291c35b/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-12-09T20:30:33.603 INFO:teuthology.orchestra.run.smithi067:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-12-09T20:30:33.672 INFO:teuthology.orchestra.run.smithi081:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-12-09T20:30:33.720 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-daemon, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse on remote rpm x86_64 2019-12-09T20:30:33.720 INFO:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/827e03c8a6b1e17d93dcaf79724437bac291c35b/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-12-09T20:30:33.778 INFO:teuthology.orchestra.run.smithi081:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-12-09T20:30:33.828 INFO:teuthology.orchestra.run.smithi067:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-12-09T20:30:33.886 INFO:teuthology.orchestra.run.smithi081.stdout:check_obsoletes = 1 2019-12-09T20:30:33.889 INFO:teuthology.orchestra.run.smithi081:> sudo yum clean all 2019-12-09T20:30:33.931 INFO:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-12-09T20:30:34.041 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2019-12-09T20:30:34.043 INFO:teuthology.orchestra.run.smithi067:> sudo yum clean all 2019-12-09T20:30:34.114 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:30:34.157 INFO:teuthology.orchestra.run.smithi081.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2019-12-09T20:30:34.158 INFO:teuthology.orchestra.run.smithi081.stdout: : lab-extras updates 2019-12-09T20:30:34.205 INFO:teuthology.orchestra.run.smithi081.stdout:Cleaning up list of fastest mirrors 2019-12-09T20:30:34.205 INFO:teuthology.orchestra.run.smithi081.stdout:Other repos take up 1.5 M of disk space (use --verbose for details) 2019-12-09T20:30:34.218 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-radosgw 2019-12-09T20:30:34.294 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:30:34.343 INFO:teuthology.orchestra.run.smithi067.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2019-12-09T20:30:34.343 INFO:teuthology.orchestra.run.smithi067.stdout: : lab-extras updates 2019-12-09T20:30:34.392 INFO:teuthology.orchestra.run.smithi067.stdout:Cleaning up list of fastest mirrors 2019-12-09T20:30:34.393 INFO:teuthology.orchestra.run.smithi067.stdout:Other repos take up 1.5 M of disk space (use --verbose for details) 2019-12-09T20:30:34.405 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2019-12-09T20:30:34.446 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:30:34.489 INFO:teuthology.orchestra.run.smithi081.stdout:Determining fastest mirrors 2019-12-09T20:30:34.662 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:30:34.704 INFO:teuthology.orchestra.run.smithi067.stdout:Determining fastest mirrors 2019-12-09T20:30:35.063 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:30:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:30:35.064 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:30:35.065 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:30:35.277 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:30:35.278 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:30:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:30:35.279 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:30:42.437 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:30:42.790 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:30:42.791 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:30:42.791 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.804 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.908 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.909 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.910 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: librados2 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.912 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: librados.so.2(LIBRADOS_14.2.0)(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.912 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.914 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libceph-common.so.2()(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.915 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.916 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.917 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:30:42.917 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-base.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.926 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.927 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: librbd1 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.928 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libcephfs2 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.929 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.930 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.931 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.933 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-12-09T20:30:42.934 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-12-09T20:30:42.934 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-12-09T20:30:42.935 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package librados2.x86_64 2:15.0.0-7936.g827e03c.el7 will be an update 2019-12-09T20:30:42.936 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2019-12-09T20:30:42.937 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package librgw2.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.938 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:30:42.939 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-common.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.943 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.944 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.945 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-rados = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.946 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.947 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.947 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.948 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libradosstriper1 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.949 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.950 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.950 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.954 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:42.954 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2019-12-09T20:30:42.954 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package libcephfs2.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.954 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-12-09T20:30:42.955 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package librbd1.x86_64 2:15.0.0-7936.g827e03c.el7 will be an update 2019-12-09T20:30:42.955 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-12-09T20:30:42.955 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-09T20:30:42.955 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-09T20:30:42.956 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:30:42.956 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-12-09T20:30:42.956 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.957 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.957 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.957 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-cephfs.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.959 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-12-09T20:30:42.959 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-rados.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.959 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-rbd.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.960 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-rgw.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:42.960 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-12-09T20:30:43.008 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:30:43.372 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:30:43.373 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:30:43.373 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.387 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.496 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.498 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.499 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: librados2 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.501 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: librados.so.2(LIBRADOS_14.2.0)(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.501 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.509 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libceph-common.so.2()(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.509 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.509 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.509 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:30:43.509 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-base.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.515 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.517 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: librbd1 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.518 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libcephfs2 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.519 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.520 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.521 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.524 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-12-09T20:30:43.525 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-12-09T20:30:43.525 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-12-09T20:30:43.525 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package librados2.x86_64 2:15.0.0-7936.g827e03c.el7 will be an update 2019-12-09T20:30:43.527 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2019-12-09T20:30:43.528 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package librgw2.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.529 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:30:43.529 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-common.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.534 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.535 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.536 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-rados = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.537 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.538 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.539 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.540 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libradosstriper1 = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.541 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.542 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.543 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.544 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:30:43.544 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2019-12-09T20:30:43.544 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libcephfs2.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.545 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-12-09T20:30:43.545 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package librbd1.x86_64 2:15.0.0-7936.g827e03c.el7 will be an update 2019-12-09T20:30:43.546 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-12-09T20:30:43.546 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-09T20:30:43.560 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-09T20:30:43.561 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:30:43.561 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-12-09T20:30:43.561 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.561 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.561 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.561 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-cephfs.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.561 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-12-09T20:30:43.562 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-rados.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.562 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-rbd.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.562 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-rgw.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:30:43.562 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-12-09T20:30:43.815 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:30:43.901 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:30:43.901 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:30:43.904 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:30:43.904 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:30:43.904 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository 2019-12-09T20:30:43.905 INFO:teuthology.orchestra.run.smithi081.stdout: Size 2019-12-09T20:30:43.905 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:30:43.905 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:30:43.905 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw x86_64 2:15.0.0-7936.g827e03c.el7 ceph 6.3 M 2019-12-09T20:30:43.905 INFO:teuthology.orchestra.run.smithi081.stdout:Installing for dependencies: 2019-12-09T20:30:43.905 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base x86_64 2:15.0.0-7936.g827e03c.el7 ceph 5.6 M 2019-12-09T20:30:43.906 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common x86_64 2:15.0.0-7936.g827e03c.el7 ceph 20 M 2019-12-09T20:30:43.906 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux x86_64 2:15.0.0-7936.g827e03c.el7 ceph 20 k 2019-12-09T20:30:43.906 INFO:teuthology.orchestra.run.smithi081.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2019-12-09T20:30:43.906 INFO:teuthology.orchestra.run.smithi081.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-12-09T20:30:43.906 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 489 k 2019-12-09T20:30:43.906 INFO:teuthology.orchestra.run.smithi081.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-12-09T20:30:43.906 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2019-12-09T20:30:43.907 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 374 k 2019-12-09T20:30:43.907 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2019-12-09T20:30:43.907 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 5.5 M 2019-12-09T20:30:43.907 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-12-09T20:30:43.907 INFO:teuthology.orchestra.run.smithi081.stdout: python-ceph-argparse x86_64 2:15.0.0-7936.g827e03c.el7 ceph 35 k 2019-12-09T20:30:43.907 INFO:teuthology.orchestra.run.smithi081.stdout: python-ceph-common x86_64 2:15.0.0-7936.g827e03c.el7 ceph 42 k 2019-12-09T20:30:43.907 INFO:teuthology.orchestra.run.smithi081.stdout: python-cephfs x86_64 2:15.0.0-7936.g827e03c.el7 ceph 102 k 2019-12-09T20:30:43.908 INFO:teuthology.orchestra.run.smithi081.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2019-12-09T20:30:43.908 INFO:teuthology.orchestra.run.smithi081.stdout: python-rados x86_64 2:15.0.0-7936.g827e03c.el7 ceph 209 k 2019-12-09T20:30:43.908 INFO:teuthology.orchestra.run.smithi081.stdout: python-rbd x86_64 2:15.0.0-7936.g827e03c.el7 ceph 199 k 2019-12-09T20:30:43.908 INFO:teuthology.orchestra.run.smithi081.stdout: python-rgw x86_64 2:15.0.0-7936.g827e03c.el7 ceph 77 k 2019-12-09T20:30:43.908 INFO:teuthology.orchestra.run.smithi081.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-12-09T20:30:43.908 INFO:teuthology.orchestra.run.smithi081.stdout:Updating for dependencies: 2019-12-09T20:30:43.908 INFO:teuthology.orchestra.run.smithi081.stdout: librados2 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 3.5 M 2019-12-09T20:30:43.909 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 1.8 M 2019-12-09T20:30:43.909 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:30:43.909 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:30:43.909 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:30:43.909 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package (+20 Dependent packages) 2019-12-09T20:30:43.910 INFO:teuthology.orchestra.run.smithi081.stdout:Upgrade ( 2 Dependent packages) 2019-12-09T20:30:43.910 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:30:43.910 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 45 M 2019-12-09T20:30:43.910 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:30:43.911 INFO:teuthology.orchestra.run.smithi081.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-09T20:30:44.461 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:30:44.551 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:30:44.551 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:30:44.554 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:30:44.555 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:30:44.555 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository 2019-12-09T20:30:44.555 INFO:teuthology.orchestra.run.smithi067.stdout: Size 2019-12-09T20:30:44.555 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:30:44.555 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:30:44.555 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:15.0.0-7936.g827e03c.el7 ceph 6.3 M 2019-12-09T20:30:44.555 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:30:44.556 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:15.0.0-7936.g827e03c.el7 ceph 5.6 M 2019-12-09T20:30:44.556 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:15.0.0-7936.g827e03c.el7 ceph 20 M 2019-12-09T20:30:44.556 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:15.0.0-7936.g827e03c.el7 ceph 20 k 2019-12-09T20:30:44.556 INFO:teuthology.orchestra.run.smithi067.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2019-12-09T20:30:44.556 INFO:teuthology.orchestra.run.smithi067.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-12-09T20:30:44.556 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 489 k 2019-12-09T20:30:44.556 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-12-09T20:30:44.557 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2019-12-09T20:30:44.557 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 374 k 2019-12-09T20:30:44.557 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2019-12-09T20:30:44.557 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 5.5 M 2019-12-09T20:30:44.557 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-12-09T20:30:44.557 INFO:teuthology.orchestra.run.smithi067.stdout: python-ceph-argparse x86_64 2:15.0.0-7936.g827e03c.el7 ceph 35 k 2019-12-09T20:30:44.557 INFO:teuthology.orchestra.run.smithi067.stdout: python-ceph-common x86_64 2:15.0.0-7936.g827e03c.el7 ceph 42 k 2019-12-09T20:30:44.558 INFO:teuthology.orchestra.run.smithi067.stdout: python-cephfs x86_64 2:15.0.0-7936.g827e03c.el7 ceph 102 k 2019-12-09T20:30:44.558 INFO:teuthology.orchestra.run.smithi067.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2019-12-09T20:30:44.558 INFO:teuthology.orchestra.run.smithi067.stdout: python-rados x86_64 2:15.0.0-7936.g827e03c.el7 ceph 209 k 2019-12-09T20:30:44.558 INFO:teuthology.orchestra.run.smithi067.stdout: python-rbd x86_64 2:15.0.0-7936.g827e03c.el7 ceph 199 k 2019-12-09T20:30:44.558 INFO:teuthology.orchestra.run.smithi067.stdout: python-rgw x86_64 2:15.0.0-7936.g827e03c.el7 ceph 77 k 2019-12-09T20:30:44.558 INFO:teuthology.orchestra.run.smithi067.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-12-09T20:30:44.558 INFO:teuthology.orchestra.run.smithi067.stdout:Updating for dependencies: 2019-12-09T20:30:44.559 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 3.5 M 2019-12-09T20:30:44.559 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:15.0.0-7936.g827e03c.el7 ceph 1.8 M 2019-12-09T20:30:44.559 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:30:44.559 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:30:44.559 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:30:44.559 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package (+20 Dependent packages) 2019-12-09T20:30:44.559 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade ( 2 Dependent packages) 2019-12-09T20:30:44.560 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:30:44.560 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 45 M 2019-12-09T20:30:44.560 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:30:44.560 INFO:teuthology.orchestra.run.smithi067.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-09T20:30:51.429 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:30:51.429 INFO:teuthology.orchestra.run.smithi081.stdout:Total 6.0 MB/s | 45 MB 00:07 2019-12-09T20:30:51.444 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:30:51.521 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:30:51.767 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:30:51.775 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:30:52.470 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:30:52.470 INFO:teuthology.orchestra.run.smithi067.stdout:Total 5.7 MB/s | 45 MB 00:07 2019-12-09T20:30:52.480 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/25 2019-12-09T20:30:52.482 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:30:52.535 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:30:52.731 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:30:52.731 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:30:52.840 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/25 2019-12-09T20:30:53.192 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 3/25 2019-12-09T20:30:53.507 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.2-1.el7.x86_64 1/25 2019-12-09T20:30:53.524 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:python-ceph-argparse-15.0.0-7936.g827e03c.el7.x86_64 4/25 2019-12-09T20:30:53.837 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 2/25 2019-12-09T20:30:53.869 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 5/25 2019-12-09T20:30:54.147 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 3/25 2019-12-09T20:30:54.238 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 6/25 2019-12-09T20:30:54.421 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:python-ceph-argparse-15.0.0-7936.g827e03c.el7.x86_64 4/25 2019-12-09T20:30:54.756 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 5/25 2019-12-09T20:30:55.045 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 7/25 2019-12-09T20:30:55.236 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 6/25 2019-12-09T20:30:55.430 INFO:teuthology.orchestra.run.smithi081.stdout: Updating : 2:librados2-15.0.0-7936.g827e03c.el7.x86_64 8/25 2019-12-09T20:30:55.967 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:python-rados-15.0.0-7936.g827e03c.el7.x86_64 9/25 2019-12-09T20:30:56.026 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 7/25 2019-12-09T20:30:56.336 INFO:teuthology.orchestra.run.smithi067.stdout: Updating : 2:librados2-15.0.0-7936.g827e03c.el7.x86_64 8/25 2019-12-09T20:30:56.825 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:python-rados-15.0.0-7936.g827e03c.el7.x86_64 9/25 2019-12-09T20:30:56.857 INFO:teuthology.orchestra.run.smithi081.stdout: Updating : 2:librbd1-15.0.0-7936.g827e03c.el7.x86_64 10/25 2019-12-09T20:30:57.190 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:librgw2-15.0.0-7936.g827e03c.el7.x86_64 11/25 2019-12-09T20:30:57.522 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:libcephfs2-15.0.0-7936.g827e03c.el7.x86_64 12/25 2019-12-09T20:30:57.696 INFO:teuthology.orchestra.run.smithi067.stdout: Updating : 2:librbd1-15.0.0-7936.g827e03c.el7.x86_64 10/25 2019-12-09T20:30:57.806 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:python-cephfs-15.0.0-7936.g827e03c.el7.x86_64 13/25 2019-12-09T20:30:58.044 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:librgw2-15.0.0-7936.g827e03c.el7.x86_64 11/25 2019-12-09T20:30:58.126 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:python-rgw-15.0.0-7936.g827e03c.el7.x86_64 14/25 2019-12-09T20:30:58.343 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:libcephfs2-15.0.0-7936.g827e03c.el7.x86_64 12/25 2019-12-09T20:30:58.504 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:python-rbd-15.0.0-7936.g827e03c.el7.x86_64 15/25 2019-12-09T20:30:58.629 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:python-cephfs-15.0.0-7936.g827e03c.el7.x86_64 13/25 2019-12-09T20:30:58.862 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:libradosstriper1-15.0.0-7936.g827e03c.el7.x86_64 16/25 2019-12-09T20:30:58.926 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:python-rgw-15.0.0-7936.g827e03c.el7.x86_64 14/25 2019-12-09T20:30:59.163 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:python-ceph-common-15.0.0-7936.g827e03c.el7.x86_64 17/25 2019-12-09T20:30:59.240 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:python-rbd-15.0.0-7936.g827e03c.el7.x86_64 15/25 2019-12-09T20:30:59.504 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 18/25 2019-12-09T20:30:59.519 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:libradosstriper1-15.0.0-7936.g827e03c.el7.x86_64 16/25 2019-12-09T20:30:59.801 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:python-ceph-common-15.0.0-7936.g827e03c.el7.x86_64 17/25 2019-12-09T20:31:00.100 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 18/25 2019-12-09T20:31:02.034 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 19/25 2019-12-09T20:31:02.575 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 19/25 2019-12-09T20:31:03.246 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 20/25 2019-12-09T20:31:03.723 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 20/25 2019-12-09T20:31:03.858 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 21/25 2019-12-09T20:31:04.249 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 21/25 2019-12-09T20:31:13.849 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-selinux-15.0.0-7936.g827e03c.el7.x86_64 22/25 2019-12-09T20:31:14.303 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 23/25 2019-12-09T20:31:14.375 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-selinux-15.0.0-7936.g827e03c.el7.x86_64 22/25 2019-12-09T20:31:14.587 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 24/25 2019-12-09T20:31:14.813 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 23/25 2019-12-09T20:31:14.906 INFO:teuthology.orchestra.run.smithi081.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 25/25 2019-12-09T20:31:15.067 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:python-rgw-15.0.0-7936.g827e03c.el7.x86_64 1/25 2019-12-09T20:31:15.083 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 24/25 2019-12-09T20:31:15.159 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 2/25 2019-12-09T20:31:15.268 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 3/25 2019-12-09T20:31:15.335 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 25/25 2019-12-09T20:31:15.405 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 4/25 2019-12-09T20:31:15.426 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:python-rgw-15.0.0-7936.g827e03c.el7.x86_64 1/25 2019-12-09T20:31:15.502 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:librbd1-15.0.0-7936.g827e03c.el7.x86_64 5/25 2019-12-09T20:31:15.521 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 2/25 2019-12-09T20:31:15.644 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-radosgw-15.0.0-7936.g827e03c.el7.x86_64 3/25 2019-12-09T20:31:15.645 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 6/25 2019-12-09T20:31:15.753 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 4/25 2019-12-09T20:31:15.770 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:python-rbd-15.0.0-7936.g827e03c.el7.x86_64 7/25 2019-12-09T20:31:15.862 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:librbd1-15.0.0-7936.g827e03c.el7.x86_64 5/25 2019-12-09T20:31:15.901 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-selinux-15.0.0-7936.g827e03c.el7.x86_64 8/25 2019-12-09T20:31:15.979 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 6/25 2019-12-09T20:31:16.021 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:python-rados-15.0.0-7936.g827e03c.el7.x86_64 9/25 2019-12-09T20:31:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:python-rbd-15.0.0-7936.g827e03c.el7.x86_64 7/25 2019-12-09T20:31:16.138 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:libradosstriper1-15.0.0-7936.g827e03c.el7.x86_64 10/25 2019-12-09T20:31:16.230 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-selinux-15.0.0-7936.g827e03c.el7.x86_64 8/25 2019-12-09T20:31:16.297 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 11/25 2019-12-09T20:31:16.339 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:python-rados-15.0.0-7936.g827e03c.el7.x86_64 9/25 2019-12-09T20:31:16.446 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:python-cephfs-15.0.0-7936.g827e03c.el7.x86_64 12/25 2019-12-09T20:31:16.448 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:libradosstriper1-15.0.0-7936.g827e03c.el7.x86_64 10/25 2019-12-09T20:31:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-base-15.0.0-7936.g827e03c.el7.x86_64 11/25 2019-12-09T20:31:16.573 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:python-ceph-argparse-15.0.0-7936.g827e03c.el7.x86_64 13/25 2019-12-09T20:31:16.674 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:python-cephfs-15.0.0-7936.g827e03c.el7.x86_64 12/25 2019-12-09T20:31:16.733 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:librgw2-15.0.0-7936.g827e03c.el7.x86_64 14/25 2019-12-09T20:31:16.758 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:python-ceph-argparse-15.0.0-7936.g827e03c.el7.x86_64 13/25 2019-12-09T20:31:16.833 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 15/25 2019-12-09T20:31:16.858 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:librgw2-15.0.0-7936.g827e03c.el7.x86_64 14/25 2019-12-09T20:31:16.925 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 16/25 2019-12-09T20:31:16.934 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 15/25 2019-12-09T20:31:17.033 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 16/25 2019-12-09T20:31:17.051 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:librados2-15.0.0-7936.g827e03c.el7.x86_64 17/25 2019-12-09T20:31:17.133 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:librados2-15.0.0-7936.g827e03c.el7.x86_64 17/25 2019-12-09T20:31:17.151 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 18/25 2019-12-09T20:31:17.227 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 18/25 2019-12-09T20:31:17.235 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:python-ceph-common-15.0.0-7936.g827e03c.el7.x86_64 19/25 2019-12-09T20:31:17.327 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:python-ceph-common-15.0.0-7936.g827e03c.el7.x86_64 19/25 2019-12-09T20:31:17.377 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 20/25 2019-12-09T20:31:17.444 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 20/25 2019-12-09T20:31:17.495 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:libcephfs2-15.0.0-7936.g827e03c.el7.x86_64 21/25 2019-12-09T20:31:17.562 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:libcephfs2-15.0.0-7936.g827e03c.el7.x86_64 21/25 2019-12-09T20:31:17.662 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 22/25 2019-12-09T20:31:17.695 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-common-15.0.0-7936.g827e03c.el7.x86_64 22/25 2019-12-09T20:31:17.788 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 23/25 2019-12-09T20:31:17.789 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 24/25 2019-12-09T20:31:17.826 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 23/25 2019-12-09T20:31:17.827 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 24/25 2019-12-09T20:31:18.212 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 25/25 2019-12-09T20:31:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:18.213 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:31:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:18.213 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:31:18.213 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: leveldb.x86_64 0:1.12.0-11.el7 2019-12-09T20:31:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-12-09T20:31:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-12-09T20:31:18.214 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-12-09T20:31:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2019-12-09T20:31:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-12-09T20:31:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: python-ceph-argparse.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: python-ceph-common.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.215 INFO:teuthology.orchestra.run.smithi067.stdout: python-cephfs.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-12-09T20:31:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: python-rados.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: python-rbd.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: python-rgw.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-12-09T20:31:18.216 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:18.217 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Updated: 2019-12-09T20:31:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: librados2.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.217 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:18.218 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:31:18.263 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 25/25 2019-12-09T20:31:18.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:18.264 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:31:18.264 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-radosgw.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.264 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:18.265 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Installed: 2019-12-09T20:31:18.265 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-base.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.265 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-common.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.266 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-selinux.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.266 INFO:teuthology.orchestra.run.smithi081.stdout: leveldb.x86_64 0:1.12.0-11.el7 2019-12-09T20:31:18.267 INFO:teuthology.orchestra.run.smithi081.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-12-09T20:31:18.268 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs2.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.268 INFO:teuthology.orchestra.run.smithi081.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-12-09T20:31:18.268 INFO:teuthology.orchestra.run.smithi081.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-12-09T20:31:18.268 INFO:teuthology.orchestra.run.smithi081.stdout: libradosstriper1.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.269 INFO:teuthology.orchestra.run.smithi081.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2019-12-09T20:31:18.269 INFO:teuthology.orchestra.run.smithi081.stdout: librgw2.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.269 INFO:teuthology.orchestra.run.smithi081.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-12-09T20:31:18.269 INFO:teuthology.orchestra.run.smithi081.stdout: python-ceph-argparse.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.270 INFO:teuthology.orchestra.run.smithi081.stdout: python-ceph-common.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.270 INFO:teuthology.orchestra.run.smithi081.stdout: python-cephfs.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.270 INFO:teuthology.orchestra.run.smithi081.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-12-09T20:31:18.270 INFO:teuthology.orchestra.run.smithi081.stdout: python-rados.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.271 INFO:teuthology.orchestra.run.smithi081.stdout: python-rbd.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.271 INFO:teuthology.orchestra.run.smithi081.stdout: python-rgw.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.271 INFO:teuthology.orchestra.run.smithi081.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-12-09T20:31:18.271 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:18.271 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Updated: 2019-12-09T20:31:18.272 INFO:teuthology.orchestra.run.smithi081.stdout: librados2.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.272 INFO:teuthology.orchestra.run.smithi081.stdout: librbd1.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:18.272 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:18.272 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:31:18.343 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2019-12-09T20:31:18.395 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-test 2019-12-09T20:31:18.643 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:18.662 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:18.681 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:18.682 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:18.682 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:18.682 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:18.683 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:18.708 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:18.709 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:18.710 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:18.710 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:18.710 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:19.315 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:19.385 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:19.639 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:31:19.639 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:19.639 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-test.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:19.656 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:19.686 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:19.687 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:19.688 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:19.688 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-12-09T20:31:19.688 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-12-09T20:31:19.689 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-12-09T20:31:19.691 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2019-12-09T20:31:19.694 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:19.694 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-12-09T20:31:19.735 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:31:19.736 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:19.736 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-test.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:19.755 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:19.786 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:19.787 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:19.788 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:19.788 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-12-09T20:31:19.789 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-12-09T20:31:19.790 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-12-09T20:31:19.792 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2019-12-09T20:31:19.794 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:19.794 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-12-09T20:31:20.051 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:31:20.176 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:31:20.269 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:20.269 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:31:20.270 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:20.270 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:20.270 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository 2019-12-09T20:31:20.271 INFO:teuthology.orchestra.run.smithi081.stdout: Size 2019-12-09T20:31:20.271 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:20.271 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:31:20.271 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test x86_64 2:15.0.0-7936.g827e03c.el7 ceph 38 M 2019-12-09T20:31:20.271 INFO:teuthology.orchestra.run.smithi081.stdout:Installing for dependencies: 2019-12-09T20:31:20.271 INFO:teuthology.orchestra.run.smithi081.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-12-09T20:31:20.272 INFO:teuthology.orchestra.run.smithi081.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-12-09T20:31:20.272 INFO:teuthology.orchestra.run.smithi081.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2019-12-09T20:31:20.272 INFO:teuthology.orchestra.run.smithi081.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2019-12-09T20:31:20.272 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:20.272 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:31:20.273 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:20.273 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package (+4 Dependent packages) 2019-12-09T20:31:20.273 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:20.273 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 38 M 2019-12-09T20:31:20.275 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 134 M 2019-12-09T20:31:20.275 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:31:20.404 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:20.404 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:31:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:20.406 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository 2019-12-09T20:31:20.406 INFO:teuthology.orchestra.run.smithi067.stdout: Size 2019-12-09T20:31:20.406 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:20.407 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:31:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:15.0.0-7936.g827e03c.el7 ceph 38 M 2019-12-09T20:31:20.407 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:31:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-12-09T20:31:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-12-09T20:31:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2019-12-09T20:31:20.407 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2019-12-09T20:31:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:20.408 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:31:20.408 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:20.408 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package (+4 Dependent packages) 2019-12-09T20:31:20.408 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:20.408 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 38 M 2019-12-09T20:31:20.408 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 134 M 2019-12-09T20:31:20.409 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:31:22.584 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:31:22.584 INFO:teuthology.orchestra.run.smithi081.stdout:Total 17 MB/s | 38 MB 00:02 2019-12-09T20:31:22.587 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:31:22.608 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:31:22.699 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:31:22.700 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:31:23.426 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/5 2019-12-09T20:31:23.586 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:31:23.587 INFO:teuthology.orchestra.run.smithi067.stdout:Total 12 MB/s | 38 MB 00:03 2019-12-09T20:31:23.587 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:31:23.618 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:31:23.671 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:31:23.672 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:31:23.725 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 2/5 2019-12-09T20:31:24.079 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 3/5 2019-12-09T20:31:24.432 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 1/5 2019-12-09T20:31:24.668 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 2/5 2019-12-09T20:31:24.939 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 3/5 2019-12-09T20:31:28.112 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : jq-1.5-1.el7.x86_64 4/5 2019-12-09T20:31:28.472 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 5/5 2019-12-09T20:31:28.589 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 1/5 2019-12-09T20:31:28.682 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 2/5 2019-12-09T20:31:28.782 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 3/5 2019-12-09T20:31:28.891 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : jq-1.5-1.el7.x86_64 4/5 2019-12-09T20:31:29.146 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.5-1.el7.x86_64 4/5 2019-12-09T20:31:29.309 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 5/5 2019-12-09T20:31:29.310 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:29.310 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:31:29.310 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-test.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:29.310 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:29.310 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Installed: 2019-12-09T20:31:29.310 INFO:teuthology.orchestra.run.smithi081.stdout: jq.x86_64 0:1.5-1.el7 oniguruma.x86_64 0:5.9.5-3.el7 2019-12-09T20:31:29.311 INFO:teuthology.orchestra.run.smithi081.stdout: socat.x86_64 0:1.7.3.2-2.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 2019-12-09T20:31:29.311 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:29.311 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:31:29.439 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph 2019-12-09T20:31:29.718 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:29.764 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:29.765 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:29.765 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 5/5 2019-12-09T20:31:29.766 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:29.766 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:29.766 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:29.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-test-15.0.0-7936.g827e03c.el7.x86_64 1/5 2019-12-09T20:31:29.981 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 2/5 2019-12-09T20:31:30.101 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 3/5 2019-12-09T20:31:30.189 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.5-1.el7.x86_64 4/5 2019-12-09T20:31:30.415 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 5/5 2019-12-09T20:31:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:30.607 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:31:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:30.607 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:30.608 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:31:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: jq.x86_64 0:1.5-1.el7 oniguruma.x86_64 0:5.9.5-3.el7 2019-12-09T20:31:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: socat.x86_64 0:1.7.3.2-2.el7 xmlstarlet.x86_64 0:1.6.1-1.el7 2019-12-09T20:31:30.608 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:30.608 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:31:30.735 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2019-12-09T20:31:30.760 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:31:30.760 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:30.761 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:30.763 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.792 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.793 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.813 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.814 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:30.814 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-mds.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:30.814 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:30.814 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.814 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.814 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.815 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.815 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.815 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:30.815 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-mon.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:30.829 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-osd.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:30.830 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:30.830 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2019-12-09T20:31:30.830 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-12-09T20:31:30.830 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-12-09T20:31:30.830 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:30.830 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:30.831 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:30.831 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:30.831 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:30.832 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-requests.noarch 0:2.6.0-7.el7_7 will be installed 2019-12-09T20:31:30.832 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-7.el7_7.noarch 2019-12-09T20:31:30.833 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-12-09T20:31:30.833 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-12-09T20:31:30.833 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-12-09T20:31:30.833 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:30.833 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-12-09T20:31:30.833 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-12-09T20:31:30.834 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-12-09T20:31:30.834 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-12-09T20:31:30.834 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-12-09T20:31:30.834 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-12-09T20:31:30.836 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-12-09T20:31:30.836 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-12-09T20:31:30.837 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-12-09T20:31:30.837 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:30.837 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-12-09T20:31:30.837 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-12-09T20:31:30.837 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-12-09T20:31:30.837 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:30.837 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-12-09T20:31:30.838 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-12-09T20:31:30.838 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:30.838 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-12-09T20:31:31.016 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:31.069 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:31.069 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:31.070 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:31.193 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:31:31.413 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:31.413 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:31:31.415 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:31.416 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:31.416 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository 2019-12-09T20:31:31.416 INFO:teuthology.orchestra.run.smithi081.stdout: Size 2019-12-09T20:31:31.416 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:31.416 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:31:31.416 INFO:teuthology.orchestra.run.smithi081.stdout: ceph x86_64 2:15.0.0-7936.g827e03c.el7 ceph 2.0 k 2019-12-09T20:31:31.416 INFO:teuthology.orchestra.run.smithi081.stdout:Installing for dependencies: 2019-12-09T20:31:31.417 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds x86_64 2:15.0.0-7936.g827e03c.el7 ceph 1.9 M 2019-12-09T20:31:31.417 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr x86_64 2:15.0.0-7936.g827e03c.el7 ceph 1.7 M 2019-12-09T20:31:31.417 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon x86_64 2:15.0.0-7936.g827e03c.el7 ceph 4.4 M 2019-12-09T20:31:31.417 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd x86_64 2:15.0.0-7936.g827e03c.el7 ceph 16 M 2019-12-09T20:31:31.417 INFO:teuthology.orchestra.run.smithi081.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2019-12-09T20:31:31.417 INFO:teuthology.orchestra.run.smithi081.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2019-12-09T20:31:31.417 INFO:teuthology.orchestra.run.smithi081.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2019-12-09T20:31:31.418 INFO:teuthology.orchestra.run.smithi081.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2019-12-09T20:31:31.418 INFO:teuthology.orchestra.run.smithi081.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2019-12-09T20:31:31.418 INFO:teuthology.orchestra.run.smithi081.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2019-12-09T20:31:31.418 INFO:teuthology.orchestra.run.smithi081.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-12-09T20:31:31.418 INFO:teuthology.orchestra.run.smithi081.stdout: python-requests noarch 2.6.0-7.el7_7 updates 95 k 2019-12-09T20:31:31.418 INFO:teuthology.orchestra.run.smithi081.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2019-12-09T20:31:31.418 INFO:teuthology.orchestra.run.smithi081.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2019-12-09T20:31:31.419 INFO:teuthology.orchestra.run.smithi081.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2019-12-09T20:31:31.419 INFO:teuthology.orchestra.run.smithi081.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2019-12-09T20:31:31.419 INFO:teuthology.orchestra.run.smithi081.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2019-12-09T20:31:31.419 INFO:teuthology.orchestra.run.smithi081.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2019-12-09T20:31:31.419 INFO:teuthology.orchestra.run.smithi081.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2019-12-09T20:31:31.419 INFO:teuthology.orchestra.run.smithi081.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-12-09T20:31:31.419 INFO:teuthology.orchestra.run.smithi081.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-12-09T20:31:31.420 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:31.420 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:31:31.420 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:31.420 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package (+21 Dependent packages) 2019-12-09T20:31:31.420 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:31.421 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 28 M 2019-12-09T20:31:31.421 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 87 M 2019-12-09T20:31:31.421 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:31:31.734 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:32.083 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:31:32.084 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:32.084 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:32.086 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.117 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.118 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.119 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: ceph-mgr = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.121 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:32.121 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-mds.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:32.132 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:32.134 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.134 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.137 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.138 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.138 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.140 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:31:32.140 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-mon.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:32.143 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-osd.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:32.148 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:32.148 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2019-12-09T20:31:32.149 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-12-09T20:31:32.150 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-12-09T20:31:32.151 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:32.151 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:32.152 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-12-09T20:31:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-requests.noarch 0:2.6.0-7.el7_7 will be installed 2019-12-09T20:31:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-7.el7_7.noarch 2019-12-09T20:31:32.157 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-12-09T20:31:32.157 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-12-09T20:31:32.157 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-12-09T20:31:32.158 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:32.158 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-12-09T20:31:32.159 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-12-09T20:31:32.159 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-12-09T20:31:32.161 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-12-09T20:31:32.161 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-12-09T20:31:32.161 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-12-09T20:31:32.162 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-12-09T20:31:32.162 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-12-09T20:31:32.162 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-12-09T20:31:32.162 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:32.162 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-12-09T20:31:32.163 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-12-09T20:31:32.163 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-12-09T20:31:32.163 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:32.164 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-12-09T20:31:32.164 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-12-09T20:31:32.164 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:32.164 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-12-09T20:31:32.545 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:31:32.774 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:31:32.774 INFO:teuthology.orchestra.run.smithi081.stdout:Total 20 MB/s | 28 MB 00:01 2019-12-09T20:31:32.775 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:32.775 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:31:32.779 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:32.779 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:32.779 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository 2019-12-09T20:31:32.779 INFO:teuthology.orchestra.run.smithi067.stdout: Size 2019-12-09T20:31:32.779 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:32.780 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:31:32.780 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:15.0.0-7936.g827e03c.el7 ceph 2.0 k 2019-12-09T20:31:32.780 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:31:32.780 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:15.0.0-7936.g827e03c.el7 ceph 1.9 M 2019-12-09T20:31:32.780 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:15.0.0-7936.g827e03c.el7 ceph 1.7 M 2019-12-09T20:31:32.780 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:15.0.0-7936.g827e03c.el7 ceph 4.4 M 2019-12-09T20:31:32.780 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:15.0.0-7936.g827e03c.el7 ceph 16 M 2019-12-09T20:31:32.781 INFO:teuthology.orchestra.run.smithi067.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2019-12-09T20:31:32.781 INFO:teuthology.orchestra.run.smithi067.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2019-12-09T20:31:32.781 INFO:teuthology.orchestra.run.smithi067.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2019-12-09T20:31:32.781 INFO:teuthology.orchestra.run.smithi067.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2019-12-09T20:31:32.781 INFO:teuthology.orchestra.run.smithi067.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2019-12-09T20:31:32.781 INFO:teuthology.orchestra.run.smithi067.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2019-12-09T20:31:32.781 INFO:teuthology.orchestra.run.smithi067.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-12-09T20:31:32.782 INFO:teuthology.orchestra.run.smithi067.stdout: python-requests noarch 2.6.0-7.el7_7 updates 95 k 2019-12-09T20:31:32.782 INFO:teuthology.orchestra.run.smithi067.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2019-12-09T20:31:32.782 INFO:teuthology.orchestra.run.smithi067.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2019-12-09T20:31:32.782 INFO:teuthology.orchestra.run.smithi067.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2019-12-09T20:31:32.782 INFO:teuthology.orchestra.run.smithi067.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2019-12-09T20:31:32.782 INFO:teuthology.orchestra.run.smithi067.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2019-12-09T20:31:32.782 INFO:teuthology.orchestra.run.smithi067.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2019-12-09T20:31:32.783 INFO:teuthology.orchestra.run.smithi067.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2019-12-09T20:31:32.783 INFO:teuthology.orchestra.run.smithi067.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-12-09T20:31:32.783 INFO:teuthology.orchestra.run.smithi067.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-12-09T20:31:32.784 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:32.784 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:31:32.784 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:32.784 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package (+21 Dependent packages) 2019-12-09T20:31:32.784 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:32.785 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 28 M 2019-12-09T20:31:32.785 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 87 M 2019-12-09T20:31:32.785 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:31:32.793 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:31:32.824 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:31:33.144 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:31:33.144 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:31:34.135 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:31:34.135 INFO:teuthology.orchestra.run.smithi067.stdout:Total 20 MB/s | 28 MB 00:01 2019-12-09T20:31:34.159 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:31:34.203 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:31:34.266 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-webob-1.2.3-7.el7.noarch 1/22 2019-12-09T20:31:34.602 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 2/22 2019-12-09T20:31:34.775 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:31:34.775 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:31:35.398 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 3/22 2019-12-09T20:31:35.841 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-mon-15.0.0-7936.g827e03c.el7.x86_64 4/22 2019-12-09T20:31:35.918 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-webob-1.2.3-7.el7.noarch 1/22 2019-12-09T20:31:36.219 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 2/22 2019-12-09T20:31:36.331 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 5/22 2019-12-09T20:31:36.715 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 6/22 2019-12-09T20:31:36.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 3/22 2019-12-09T20:31:37.209 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 7/22 2019-12-09T20:31:37.329 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-mon-15.0.0-7936.g827e03c.el7.x86_64 4/22 2019-12-09T20:31:37.794 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 5/22 2019-12-09T20:31:37.816 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 8/22 2019-12-09T20:31:38.123 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 6/22 2019-12-09T20:31:38.388 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-mds-15.0.0-7936.g827e03c.el7.x86_64 9/22 2019-12-09T20:31:38.553 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 7/22 2019-12-09T20:31:38.725 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 10/22 2019-12-09T20:31:39.106 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 8/22 2019-12-09T20:31:39.143 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 11/22 2019-12-09T20:31:39.487 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-mako-0.8.1-2.el7.noarch 12/22 2019-12-09T20:31:39.740 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-mds-15.0.0-7936.g827e03c.el7.x86_64 9/22 2019-12-09T20:31:39.855 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 13/22 2019-12-09T20:31:40.008 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 10/22 2019-12-09T20:31:40.196 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 14/22 2019-12-09T20:31:40.398 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 11/22 2019-12-09T20:31:40.726 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-mako-0.8.1-2.el7.noarch 12/22 2019-12-09T20:31:41.039 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 13/22 2019-12-09T20:31:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 14/22 2019-12-09T20:31:42.100 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-requests-2.6.0-7.el7_7.noarch 15/22 2019-12-09T20:31:42.553 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-osd-15.0.0-7936.g827e03c.el7.x86_64 16/22 2019-12-09T20:31:42.841 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python2-six-1.9.0-0.el7.noarch 17/22 2019-12-09T20:31:43.147 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 18/22 2019-12-09T20:31:43.262 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-requests-2.6.0-7.el7_7.noarch 15/22 2019-12-09T20:31:43.590 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 19/22 2019-12-09T20:31:43.609 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-osd-15.0.0-7936.g827e03c.el7.x86_64 16/22 2019-12-09T20:31:43.847 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python2-six-1.9.0-0.el7.noarch 17/22 2019-12-09T20:31:44.128 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 18/22 2019-12-09T20:31:44.293 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 20/22 2019-12-09T20:31:44.532 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 19/22 2019-12-09T20:31:44.768 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 21/22 2019-12-09T20:31:45.043 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 22/22 2019-12-09T20:31:45.177 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 1/22 2019-12-09T20:31:45.318 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 2/22 2019-12-09T20:31:45.443 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 3/22 2019-12-09T20:31:45.535 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 20/22 2019-12-09T20:31:45.576 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 4/22 2019-12-09T20:31:45.692 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-osd-15.0.0-7936.g827e03c.el7.x86_64 5/22 2019-12-09T20:31:45.820 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 6/22 2019-12-09T20:31:45.891 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 21/22 2019-12-09T20:31:45.962 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 7/22 2019-12-09T20:31:46.054 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 8/22 2019-12-09T20:31:46.132 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 22/22 2019-12-09T20:31:46.188 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-requests-2.6.0-7.el7_7.noarch 9/22 2019-12-09T20:31:46.231 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 1/22 2019-12-09T20:31:46.272 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 10/22 2019-12-09T20:31:46.316 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 2/22 2019-12-09T20:31:46.372 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 11/22 2019-12-09T20:31:46.440 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 3/22 2019-12-09T20:31:46.506 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 12/22 2019-12-09T20:31:46.549 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 4/22 2019-12-09T20:31:46.624 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 13/22 2019-12-09T20:31:46.650 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-osd-15.0.0-7936.g827e03c.el7.x86_64 5/22 2019-12-09T20:31:46.757 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 14/22 2019-12-09T20:31:46.768 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 6/22 2019-12-09T20:31:46.864 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 15/22 2019-12-09T20:31:46.917 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 7/22 2019-12-09T20:31:47.017 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 16/22 2019-12-09T20:31:47.035 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 8/22 2019-12-09T20:31:47.142 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-mds-15.0.0-7936.g827e03c.el7.x86_64 17/22 2019-12-09T20:31:47.160 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-requests-2.6.0-7.el7_7.noarch 9/22 2019-12-09T20:31:47.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 10/22 2019-12-09T20:31:47.301 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 18/22 2019-12-09T20:31:47.386 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 11/22 2019-12-09T20:31:47.459 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 19/22 2019-12-09T20:31:47.505 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 12/22 2019-12-09T20:31:47.585 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 20/22 2019-12-09T20:31:47.628 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 13/22 2019-12-09T20:31:47.704 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 14/22 2019-12-09T20:31:47.780 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-15.0.0-7936.g827e03c.el7.x86_64 15/22 2019-12-09T20:31:47.905 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 16/22 2019-12-09T20:31:48.005 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-mds-15.0.0-7936.g827e03c.el7.x86_64 17/22 2019-12-09T20:31:48.058 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 21/22 2019-12-09T20:31:48.098 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 18/22 2019-12-09T20:31:48.223 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 19/22 2019-12-09T20:31:48.315 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 20/22 2019-12-09T20:31:48.424 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 21/22 2019-12-09T20:31:48.581 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-mon-15.0.0-7936.g827e03c.el7.x86_64 22/22 2019-12-09T20:31:48.581 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:48.582 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:31:48.582 INFO:teuthology.orchestra.run.smithi081.stdout: ceph.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.582 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:48.582 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Installed: 2019-12-09T20:31:48.582 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mds.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.582 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.582 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mon.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.583 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-osd.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.583 INFO:teuthology.orchestra.run.smithi081.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2019-12-09T20:31:48.583 INFO:teuthology.orchestra.run.smithi081.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-12-09T20:31:48.583 INFO:teuthology.orchestra.run.smithi081.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-12-09T20:31:48.583 INFO:teuthology.orchestra.run.smithi081.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-12-09T20:31:48.583 INFO:teuthology.orchestra.run.smithi081.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-12-09T20:31:48.583 INFO:teuthology.orchestra.run.smithi081.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-12-09T20:31:48.584 INFO:teuthology.orchestra.run.smithi081.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-12-09T20:31:48.584 INFO:teuthology.orchestra.run.smithi081.stdout: python-requests.noarch 0:2.6.0-7.el7_7 2019-12-09T20:31:48.584 INFO:teuthology.orchestra.run.smithi081.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-12-09T20:31:48.584 INFO:teuthology.orchestra.run.smithi081.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-12-09T20:31:48.585 INFO:teuthology.orchestra.run.smithi081.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-12-09T20:31:48.585 INFO:teuthology.orchestra.run.smithi081.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-12-09T20:31:48.585 INFO:teuthology.orchestra.run.smithi081.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-12-09T20:31:48.585 INFO:teuthology.orchestra.run.smithi081.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-12-09T20:31:48.585 INFO:teuthology.orchestra.run.smithi081.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-12-09T20:31:48.586 INFO:teuthology.orchestra.run.smithi081.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-12-09T20:31:48.586 INFO:teuthology.orchestra.run.smithi081.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-12-09T20:31:48.586 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:48.587 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:31:48.715 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-daemon 2019-12-09T20:31:48.952 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-mon-15.0.0-7936.g827e03c.el7.x86_64 22/22 2019-12-09T20:31:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:48.953 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:31:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: ceph.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:48.953 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:31:48.953 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2019-12-09T20:31:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-12-09T20:31:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-12-09T20:31:48.954 INFO:teuthology.orchestra.run.smithi067.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-12-09T20:31:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-12-09T20:31:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-12-09T20:31:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-12-09T20:31:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: python-requests.noarch 0:2.6.0-7.el7_7 2019-12-09T20:31:48.955 INFO:teuthology.orchestra.run.smithi067.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-12-09T20:31:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-12-09T20:31:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-12-09T20:31:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-12-09T20:31:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-12-09T20:31:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-12-09T20:31:48.956 INFO:teuthology.orchestra.run.smithi067.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-12-09T20:31:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-12-09T20:31:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-12-09T20:31:48.957 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:48.957 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:31:49.013 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:49.060 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:49.061 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:49.061 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:49.061 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:49.062 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:49.084 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-daemon 2019-12-09T20:31:49.374 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:49.420 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:49.421 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:49.422 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:49.714 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:50.058 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:31:50.058 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:50.058 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-daemon.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:50.094 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:50.423 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:31:50.445 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:31:50.445 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:50.445 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-daemon.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:50.644 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:50.644 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:31:50.645 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:50.645 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository 2019-12-09T20:31:50.645 INFO:teuthology.orchestra.run.smithi081.stdout: Size 2019-12-09T20:31:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:50.645 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:31:50.646 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-daemon x86_64 2:15.0.0-7936.g827e03c.el7 ceph 17 k 2019-12-09T20:31:50.646 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:50.646 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:31:50.646 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:50.646 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2019-12-09T20:31:50.646 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:50.646 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 17 k 2019-12-09T20:31:50.647 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 68 k 2019-12-09T20:31:50.647 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:31:50.831 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:31:51.061 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:51.061 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:31:51.062 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:51.062 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:51.062 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository 2019-12-09T20:31:51.062 INFO:teuthology.orchestra.run.smithi067.stdout: Size 2019-12-09T20:31:51.062 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:51.063 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:31:51.063 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-daemon x86_64 2:15.0.0-7936.g827e03c.el7 ceph 17 k 2019-12-09T20:31:51.063 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:51.063 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:31:51.063 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:51.063 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2019-12-09T20:31:51.063 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:51.064 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 17 k 2019-12-09T20:31:51.064 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 68 k 2019-12-09T20:31:51.064 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:31:51.198 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:31:51.206 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:31:51.208 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:31:51.208 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:31:51.618 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:31:51.627 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:31:51.629 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:31:51.629 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:31:52.038 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-daemon-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:31:52.435 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-daemon-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:31:52.550 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-daemon-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:31:52.550 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:52.551 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:31:52.551 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-daemon.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:52.551 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:52.551 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:31:52.671 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr 2019-12-09T20:31:52.839 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-daemon-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:31:52.840 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:52.840 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:31:52.840 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-daemon.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:31:52.840 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:52.841 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:31:52.899 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:52.937 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:52.938 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:52.938 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:52.938 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:52.939 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:52.966 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2019-12-09T20:31:53.248 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:53.293 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:53.295 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:53.295 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:53.295 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:53.295 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:53.585 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:53.911 INFO:teuthology.orchestra.run.smithi081.stdout:Package 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 already installed and latest version 2019-12-09T20:31:53.911 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do 2019-12-09T20:31:53.967 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:53.971 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-dashboard 2019-12-09T20:31:54.217 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:54.261 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:54.262 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:54.263 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:54.317 INFO:teuthology.orchestra.run.smithi067.stdout:Package 2:ceph-mgr-15.0.0-7936.g827e03c.el7.x86_64 already installed and latest version 2019-12-09T20:31:54.317 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do 2019-12-09T20:31:54.384 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2019-12-09T20:31:54.615 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:31:54.657 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:31:54.658 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:31:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:31:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:31:54.659 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:31:54.918 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:55.252 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:31:55.253 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:55.253 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:55.256 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-mgr-dashboard-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:31:55.285 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:31:55.287 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:55.287 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:55.287 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-12-09T20:31:55.287 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-12-09T20:31:55.292 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:31:55.292 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-12-09T20:31:55.325 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:31:55.649 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:31:55.676 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:31:55.676 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:55.676 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:55.678 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-7936.g827e03c.el7 for package: 2:ceph-mgr-dashboard-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:31:55.709 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:31:55.710 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:55.711 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:31:55.711 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-12-09T20:31:55.711 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-12-09T20:31:55.712 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:31:55.712 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-12-09T20:31:55.870 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:55.870 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:31:55.871 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:55.871 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:55.871 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2019-12-09T20:31:55.871 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:55.871 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:31:55.872 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 3.9 M 2019-12-09T20:31:55.872 INFO:teuthology.orchestra.run.smithi081.stdout:Installing for dependencies: 2019-12-09T20:31:55.872 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 18 k 2019-12-09T20:31:55.872 INFO:teuthology.orchestra.run.smithi081.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-12-09T20:31:55.873 INFO:teuthology.orchestra.run.smithi081.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-12-09T20:31:55.874 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:55.874 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:31:55.874 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:31:55.874 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package (+3 Dependent packages) 2019-12-09T20:31:55.874 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:31:55.874 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 4.6 M 2019-12-09T20:31:55.875 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 20 M 2019-12-09T20:31:55.875 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:31:56.098 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:31:56.331 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:56.331 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:31:56.333 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:56.333 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:56.333 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2019-12-09T20:31:56.333 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:56.333 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:31:56.333 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 3.9 M 2019-12-09T20:31:56.334 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:31:56.334 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 18 k 2019-12-09T20:31:56.334 INFO:teuthology.orchestra.run.smithi067.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-12-09T20:31:56.334 INFO:teuthology.orchestra.run.smithi067.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-12-09T20:31:56.334 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:56.334 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:31:56.334 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:31:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package (+3 Dependent packages) 2019-12-09T20:31:56.335 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:31:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 4.6 M 2019-12-09T20:31:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 20 M 2019-12-09T20:31:56.335 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:31:57.371 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:31:57.371 INFO:teuthology.orchestra.run.smithi067.stdout:Total 4.4 MB/s | 4.6 MB 00:01 2019-12-09T20:31:57.377 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:31:57.385 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:31:57.668 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:31:57.668 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:31:58.165 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:31:58.165 INFO:teuthology.orchestra.run.smithi081.stdout:Total 2.0 MB/s | 4.6 MB 00:02 2019-12-09T20:31:58.172 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:31:58.182 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:31:58.320 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-7936.g827e03c.el7.noarch 1/4 2019-12-09T20:31:58.525 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:31:58.525 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:31:58.759 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 2/4 2019-12-09T20:31:59.590 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-7936.g827e03c.el7.noarch 1/4 2019-12-09T20:31:59.907 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-routes-1.13-2.el7.noarch 3/4 2019-12-09T20:32:00.079 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 2/4 2019-12-09T20:32:00.769 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-7936.g827e03c.el7.noarch 4/4 2019-12-09T20:32:00.878 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 1/4 2019-12-09T20:32:00.978 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-7936.g827e03c.el7.noarch 2/4 2019-12-09T20:32:01.053 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-7936.g827e03c.el7.noarch 3/4 2019-12-09T20:32:01.192 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-routes-1.13-2.el7.noarch 3/4 2019-12-09T20:32:01.514 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-routes-1.13-2.el7.noarch 4/4 2019-12-09T20:32:01.514 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:01.514 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:32:01.514 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:01.514 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:01.515 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:32:01.515 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:01.515 INFO:teuthology.orchestra.run.smithi067.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-12-09T20:32:01.515 INFO:teuthology.orchestra.run.smithi067.stdout: python-routes.noarch 0:1.13-2.el7 2019-12-09T20:32:01.515 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:01.515 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:32:01.632 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-7936.g827e03c.el7.noarch 4/4 2019-12-09T20:32:01.633 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-cloud 2019-12-09T20:32:01.807 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 1/4 2019-12-09T20:32:01.870 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:01.915 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:01.916 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:01.916 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:01.916 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:01.917 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:01.932 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-7936.g827e03c.el7.noarch 2/4 2019-12-09T20:32:02.074 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-7936.g827e03c.el7.noarch 3/4 2019-12-09T20:32:02.560 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-routes-1.13-2.el7.noarch 4/4 2019-12-09T20:32:02.560 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:02.560 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:32:02.560 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:02.561 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:02.561 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Installed: 2019-12-09T20:32:02.561 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:02.561 INFO:teuthology.orchestra.run.smithi081.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-12-09T20:32:02.561 INFO:teuthology.orchestra.run.smithi081.stdout: python-routes.noarch 0:1.13-2.el7 2019-12-09T20:32:02.561 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:02.562 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:32:02.585 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:02.723 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-diskprediction-cloud 2019-12-09T20:32:02.928 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:02.933 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:32:02.934 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:02.934 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:02.966 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:02.967 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:02.968 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:02.968 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:02.968 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:03.320 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:03.550 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:03.550 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:32:03.551 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:03.551 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:03.551 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2019-12-09T20:32:03.551 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:03.551 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:32:03.551 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-cloud 2019-12-09T20:32:03.551 INFO:teuthology.orchestra.run.smithi067.stdout: noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 82 k 2019-12-09T20:32:03.552 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:03.552 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:32:03.552 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:03.552 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2019-12-09T20:32:03.552 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:03.552 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 82 k 2019-12-09T20:32:03.552 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 411 k 2019-12-09T20:32:03.553 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:32:03.578 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:03.901 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:32:03.901 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:03.902 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:04.248 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:32:04.250 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:32:04.265 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:04.296 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:32:04.297 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:32:04.484 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:04.484 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:32:04.485 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:04.485 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:04.485 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2019-12-09T20:32:04.486 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:04.486 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:32:04.486 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-cloud 2019-12-09T20:32:04.486 INFO:teuthology.orchestra.run.smithi081.stdout: noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 82 k 2019-12-09T20:32:04.486 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:04.486 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:32:04.486 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:04.487 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2019-12-09T20:32:04.487 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:04.487 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 82 k 2019-12-09T20:32:04.487 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 411 k 2019-12-09T20:32:04.487 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:32:04.879 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:32:04.883 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:32:04.937 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:32:04.938 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:32:04.978 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-7936.g827e03c.el7.n 1/1 2019-12-09T20:32:05.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-7936.g827e03c.el7.n 1/1 2019-12-09T20:32:05.461 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:05.461 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:32:05.461 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:05.461 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:05.462 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:32:05.575 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-7936.g827e03c.el7.n 1/1 2019-12-09T20:32:05.583 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2019-12-09T20:32:05.817 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:05.862 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:05.863 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:05.864 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:05.864 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:05.864 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:06.097 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-7936.g827e03c.el7.n 1/1 2019-12-09T20:32:06.097 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:06.097 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:32:06.098 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:06.098 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:06.098 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:32:06.217 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-diskprediction-local 2019-12-09T20:32:06.395 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:06.433 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:06.434 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:06.434 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:06.434 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:06.435 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:06.531 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:06.881 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:32:06.881 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:06.881 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:06.883 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:32:06.915 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:06.915 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-12-09T20:32:06.922 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-12-09T20:32:06.923 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-09T20:32:06.925 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-09T20:32:06.925 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:06.926 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-12-09T20:32:06.926 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-12-09T20:32:06.927 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-09T20:32:06.929 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-09T20:32:06.929 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:06.930 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-12-09T20:32:07.033 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:07.311 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:07.354 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:32:07.355 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:07.355 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:07.357 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:32:07.387 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:07.387 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-12-09T20:32:07.393 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-12-09T20:32:07.394 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-09T20:32:07.396 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-09T20:32:07.396 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:07.397 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-12-09T20:32:07.397 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-12-09T20:32:07.398 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-09T20:32:07.400 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-09T20:32:07.400 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:07.400 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-12-09T20:32:07.549 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:07.549 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:32:07.551 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:07.551 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:07.551 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2019-12-09T20:32:07.551 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:07.551 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:32:07.552 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local 2019-12-09T20:32:07.552 INFO:teuthology.orchestra.run.smithi067.stdout: noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 8.6 M 2019-12-09T20:32:07.552 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:32:07.552 INFO:teuthology.orchestra.run.smithi067.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2019-12-09T20:32:07.552 INFO:teuthology.orchestra.run.smithi067.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2019-12-09T20:32:07.552 INFO:teuthology.orchestra.run.smithi067.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2019-12-09T20:32:07.553 INFO:teuthology.orchestra.run.smithi067.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2019-12-09T20:32:07.553 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:07.553 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:32:07.553 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:07.553 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package (+4 Dependent packages) 2019-12-09T20:32:07.553 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:07.553 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 19 M 2019-12-09T20:32:07.554 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 104 M 2019-12-09T20:32:07.554 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:32:07.760 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:07.980 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:07.981 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:32:07.981 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:07.982 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:07.982 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2019-12-09T20:32:07.982 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:07.982 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:32:07.982 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local 2019-12-09T20:32:07.982 INFO:teuthology.orchestra.run.smithi081.stdout: noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 8.6 M 2019-12-09T20:32:07.982 INFO:teuthology.orchestra.run.smithi081.stdout:Installing for dependencies: 2019-12-09T20:32:07.983 INFO:teuthology.orchestra.run.smithi081.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2019-12-09T20:32:07.983 INFO:teuthology.orchestra.run.smithi081.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2019-12-09T20:32:07.983 INFO:teuthology.orchestra.run.smithi081.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2019-12-09T20:32:07.983 INFO:teuthology.orchestra.run.smithi081.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2019-12-09T20:32:07.983 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:07.983 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:32:07.983 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:07.984 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package (+4 Dependent packages) 2019-12-09T20:32:07.984 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:07.984 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 19 M 2019-12-09T20:32:07.984 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 104 M 2019-12-09T20:32:07.984 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:32:08.815 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:08.816 INFO:teuthology.orchestra.run.smithi081.stdout:Total 23 MB/s | 19 MB 00:00 2019-12-09T20:32:08.828 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:32:08.839 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:08.839 INFO:teuthology.orchestra.run.smithi067.stdout:Total 15 MB/s | 19 MB 00:01 2019-12-09T20:32:08.852 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:32:08.859 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:32:08.886 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:32:08.982 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:32:08.982 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:32:09.028 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:32:09.028 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:32:09.862 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 1/5 2019-12-09T20:32:09.923 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 1/5 2019-12-09T20:32:10.414 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 2/5 2019-12-09T20:32:10.590 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 2/5 2019-12-09T20:32:12.170 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-12-09T20:32:12.344 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-12-09T20:32:13.850 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : scipy-0.12.1-6.el7.x86_64 4/5 2019-12-09T20:32:14.056 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : scipy-0.12.1-6.el7.x86_64 4/5 2019-12-09T20:32:14.183 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-7936.g827e03c.el7.n 5/5 2019-12-09T20:32:14.321 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 1/5 2019-12-09T20:32:14.379 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-7936.g827e03c.el7.n 5/5 2019-12-09T20:32:14.472 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-7936.g827e03c.el7.n 2/5 2019-12-09T20:32:14.506 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 1/5 2019-12-09T20:32:14.597 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-12-09T20:32:14.617 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-7936.g827e03c.el7.n 2/5 2019-12-09T20:32:14.709 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 3/5 2019-12-09T20:32:14.730 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 4/5 2019-12-09T20:32:14.793 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 4/5 2019-12-09T20:32:15.253 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 5/5 2019-12-09T20:32:15.253 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:15.253 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:32:15.253 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:15.253 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:15.253 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Installed: 2019-12-09T20:32:15.254 INFO:teuthology.orchestra.run.smithi081.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 scipy.x86_64 0:0.12.1-6.el7 2019-12-09T20:32:15.254 INFO:teuthology.orchestra.run.smithi081.stdout: suitesparse.x86_64 0:4.0.2-10.el7 tbb.x86_64 0:4.1-9.20130314.el7 2019-12-09T20:32:15.254 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:15.254 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:32:15.382 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-rook 2019-12-09T20:32:15.616 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:15.657 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:15.658 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:15.720 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 5/5 2019-12-09T20:32:15.721 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:15.721 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:32:15.721 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:15.722 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:15.722 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:32:15.722 INFO:teuthology.orchestra.run.smithi067.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 scipy.x86_64 0:0.12.1-6.el7 2019-12-09T20:32:15.722 INFO:teuthology.orchestra.run.smithi067.stdout: suitesparse.x86_64 0:4.0.2-10.el7 tbb.x86_64 0:4.1-9.20130314.el7 2019-12-09T20:32:15.723 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:15.723 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:32:15.851 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2019-12-09T20:32:16.091 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:16.134 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:16.135 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:16.136 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:16.137 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:16.303 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:16.635 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:32:16.636 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:16.636 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:16.637 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:32:16.669 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:16.669 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python2-kubernetes.noarch 0:10.0.1-1.el7 will be installed 2019-12-09T20:32:16.672 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-09T20:32:16.672 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-09T20:32:16.673 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-09T20:32:16.674 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-09T20:32:16.675 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:16.676 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-12-09T20:32:16.676 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-12-09T20:32:16.677 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-12-09T20:32:16.677 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-09T20:32:16.677 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-09T20:32:16.683 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-09T20:32:16.683 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-12-09T20:32:16.701 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-12-09T20:32:16.701 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:16.701 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-12-09T20:32:16.702 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-12-09T20:32:16.702 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-12-09T20:32:16.702 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-12-09T20:32:16.702 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-12-09T20:32:16.702 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:16.703 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-12-09T20:32:16.801 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:17.044 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:17.149 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:32:17.149 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:17.150 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:17.152 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:32:17.182 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:17.183 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python2-kubernetes.noarch 0:10.0.1-1.el7 will be installed 2019-12-09T20:32:17.185 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-09T20:32:17.186 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-09T20:32:17.187 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-09T20:32:17.188 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-09T20:32:17.189 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:17.189 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-12-09T20:32:17.190 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-12-09T20:32:17.190 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-12-09T20:32:17.191 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-09T20:32:17.191 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-09T20:32:17.192 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-09T20:32:17.193 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-12-09T20:32:17.193 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-12-09T20:32:17.194 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:17.195 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-12-09T20:32:17.195 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-12-09T20:32:17.195 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-12-09T20:32:17.196 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-12-09T20:32:17.196 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-12-09T20:32:17.196 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:17.197 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-12-09T20:32:17.266 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:17.266 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:32:17.272 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:17.272 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:17.272 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2019-12-09T20:32:17.272 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:17.272 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:32:17.272 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 23 k 2019-12-09T20:32:17.273 INFO:teuthology.orchestra.run.smithi081.stdout:Installing for dependencies: 2019-12-09T20:32:17.273 INFO:teuthology.orchestra.run.smithi081.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-12-09T20:32:17.273 INFO:teuthology.orchestra.run.smithi081.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2019-12-09T20:32:17.273 INFO:teuthology.orchestra.run.smithi081.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2019-12-09T20:32:17.273 INFO:teuthology.orchestra.run.smithi081.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-12-09T20:32:17.273 INFO:teuthology.orchestra.run.smithi081.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-12-09T20:32:17.273 INFO:teuthology.orchestra.run.smithi081.stdout: python2-kubernetes noarch 10.0.1-1.el7 epel 1.1 M 2019-12-09T20:32:17.274 INFO:teuthology.orchestra.run.smithi081.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2019-12-09T20:32:17.274 INFO:teuthology.orchestra.run.smithi081.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2019-12-09T20:32:17.274 INFO:teuthology.orchestra.run.smithi081.stdout: python2-requests-oauthlib noarch 0.8.0-5.el7 base 35 k 2019-12-09T20:32:17.274 INFO:teuthology.orchestra.run.smithi081.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-12-09T20:32:17.274 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:17.274 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:32:17.274 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:17.275 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package (+10 Dependent packages) 2019-12-09T20:32:17.275 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:17.275 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 1.6 M 2019-12-09T20:32:17.275 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 24 M 2019-12-09T20:32:17.275 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:32:17.585 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:17.816 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:17.817 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:32:17.817 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:17.818 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:17.818 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2019-12-09T20:32:17.818 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:17.818 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:32:17.818 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 23 k 2019-12-09T20:32:17.818 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:32:17.818 INFO:teuthology.orchestra.run.smithi067.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-12-09T20:32:17.819 INFO:teuthology.orchestra.run.smithi067.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2019-12-09T20:32:17.819 INFO:teuthology.orchestra.run.smithi067.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2019-12-09T20:32:17.819 INFO:teuthology.orchestra.run.smithi067.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-12-09T20:32:17.819 INFO:teuthology.orchestra.run.smithi067.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-12-09T20:32:17.819 INFO:teuthology.orchestra.run.smithi067.stdout: python2-kubernetes noarch 10.0.1-1.el7 epel 1.1 M 2019-12-09T20:32:17.819 INFO:teuthology.orchestra.run.smithi067.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2019-12-09T20:32:17.819 INFO:teuthology.orchestra.run.smithi067.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2019-12-09T20:32:17.820 INFO:teuthology.orchestra.run.smithi067.stdout: python2-requests-oauthlib noarch 0.8.0-5.el7 base 35 k 2019-12-09T20:32:17.820 INFO:teuthology.orchestra.run.smithi067.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-12-09T20:32:17.820 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:17.820 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:32:17.820 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:17.820 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package (+10 Dependent packages) 2019-12-09T20:32:17.820 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:17.821 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 1.6 M 2019-12-09T20:32:17.821 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 24 M 2019-12-09T20:32:17.821 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:32:18.435 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:18.436 INFO:teuthology.orchestra.run.smithi081.stdout:Total 1.4 MB/s | 1.6 MB 00:01 2019-12-09T20:32:18.452 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:32:18.466 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:32:18.540 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:32:18.540 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:32:19.048 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:19.048 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.3 MB/s | 1.6 MB 00:01 2019-12-09T20:32:19.063 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:32:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:32:19.139 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:32:19.139 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:32:19.323 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 1/11 2019-12-09T20:32:19.801 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 2/11 2019-12-09T20:32:19.836 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 1/11 2019-12-09T20:32:20.167 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 3/11 2019-12-09T20:32:20.516 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 2/11 2019-12-09T20:32:20.532 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 4/11 2019-12-09T20:32:20.809 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 3/11 2019-12-09T20:32:20.879 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 5/11 2019-12-09T20:32:21.085 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 4/11 2019-12-09T20:32:21.250 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-12-09T20:32:21.402 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 5/11 2019-12-09T20:32:21.629 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 7/11 2019-12-09T20:32:21.777 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-12-09T20:32:21.968 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 8/11 2019-12-09T20:32:22.122 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 7/11 2019-12-09T20:32:22.451 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 8/11 2019-12-09T20:32:23.281 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 9/11 2019-12-09T20:32:23.755 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 9/11 2019-12-09T20:32:23.815 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python2-kubernetes-10.0.1-1.el7.noarch 10/11 2019-12-09T20:32:24.121 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python2-kubernetes-10.0.1-1.el7.noarch 10/11 2019-12-09T20:32:24.157 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-mgr-rook-15.0.0-7936.g827e03c.el7.noarch 11/11 2019-12-09T20:32:24.268 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 1/11 2019-12-09T20:32:24.368 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-7936.g827e03c.el7.noarch 2/11 2019-12-09T20:32:24.382 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-mgr-rook-15.0.0-7936.g827e03c.el7.noarch 11/11 2019-12-09T20:32:24.519 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 3/11 2019-12-09T20:32:24.547 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 1/11 2019-12-09T20:32:24.628 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 4/11 2019-12-09T20:32:24.681 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-7936.g827e03c.el7.noarch 2/11 2019-12-09T20:32:24.778 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 5/11 2019-12-09T20:32:24.798 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 3/11 2019-12-09T20:32:24.890 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 4/11 2019-12-09T20:32:24.903 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-12-09T20:32:25.024 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 5/11 2019-12-09T20:32:25.046 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 7/11 2019-12-09T20:32:25.130 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 6/11 2019-12-09T20:32:25.180 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 8/11 2019-12-09T20:32:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 7/11 2019-12-09T20:32:25.313 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python2-kubernetes-10.0.1-1.el7.noarch 9/11 2019-12-09T20:32:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 8/11 2019-12-09T20:32:25.393 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python2-kubernetes-10.0.1-1.el7.noarch 9/11 2019-12-09T20:32:25.414 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 10/11 2019-12-09T20:32:25.468 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 10/11 2019-12-09T20:32:25.958 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 11/11 2019-12-09T20:32:25.958 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:25.958 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:32:25.958 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-rook.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:25.959 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:25.959 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Installed: 2019-12-09T20:32:25.959 INFO:teuthology.orchestra.run.smithi081.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-12-09T20:32:25.959 INFO:teuthology.orchestra.run.smithi081.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-12-09T20:32:25.959 INFO:teuthology.orchestra.run.smithi081.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-12-09T20:32:25.959 INFO:teuthology.orchestra.run.smithi081.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-12-09T20:32:25.959 INFO:teuthology.orchestra.run.smithi081.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-12-09T20:32:25.960 INFO:teuthology.orchestra.run.smithi081.stdout: python2-kubernetes.noarch 0:10.0.1-1.el7 2019-12-09T20:32:25.960 INFO:teuthology.orchestra.run.smithi081.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-12-09T20:32:25.960 INFO:teuthology.orchestra.run.smithi081.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-12-09T20:32:25.960 INFO:teuthology.orchestra.run.smithi081.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-12-09T20:32:25.960 INFO:teuthology.orchestra.run.smithi081.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-12-09T20:32:25.960 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:25.960 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:32:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 11/11 2019-12-09T20:32:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:25.997 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:32:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:25.997 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:25.997 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:32:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-12-09T20:32:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-12-09T20:32:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-12-09T20:32:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-12-09T20:32:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-12-09T20:32:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: python2-kubernetes.noarch 0:10.0.1-1.el7 2019-12-09T20:32:25.998 INFO:teuthology.orchestra.run.smithi067.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-12-09T20:32:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-12-09T20:32:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-12-09T20:32:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-12-09T20:32:25.999 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:25.999 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:32:26.078 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-mgr-ssh 2019-12-09T20:32:26.123 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-ssh 2019-12-09T20:32:26.295 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:26.338 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:26.339 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:26.340 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:26.340 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:26.340 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:26.374 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:26.417 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:26.417 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:26.418 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:26.418 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:26.419 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:26.988 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:27.085 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:27.313 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:32:27.314 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:27.314 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:27.316 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:32:27.346 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:27.347 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-12-09T20:32:27.347 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-12-09T20:32:27.348 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:27.348 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-12-09T20:32:27.431 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:32:27.431 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:27.431 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:27.434 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-7936.g827e03c.el7.noarch 2019-12-09T20:32:27.476 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:27.477 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-12-09T20:32:27.477 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-12-09T20:32:27.478 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:27.478 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-12-09T20:32:27.711 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:27.866 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:27.935 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:27.935 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:32:27.936 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:27.936 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:27.936 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2019-12-09T20:32:27.936 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:27.936 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:32:27.936 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-ssh noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 33 k 2019-12-09T20:32:27.936 INFO:teuthology.orchestra.run.smithi081.stdout:Installing for dependencies: 2019-12-09T20:32:27.937 INFO:teuthology.orchestra.run.smithi081.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-12-09T20:32:27.937 INFO:teuthology.orchestra.run.smithi081.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-12-09T20:32:27.937 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:27.937 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:32:27.937 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:27.937 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package (+2 Dependent packages) 2019-12-09T20:32:27.937 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:27.938 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 316 k 2019-12-09T20:32:27.938 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 1.1 M 2019-12-09T20:32:27.938 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:32:28.100 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:28.100 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:32:28.101 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:28.101 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2019-12-09T20:32:28.101 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:28.102 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:32:28.102 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-ssh noarch 2:15.0.0-7936.g827e03c.el7 ceph-noarch 33 k 2019-12-09T20:32:28.102 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:32:28.102 INFO:teuthology.orchestra.run.smithi067.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-12-09T20:32:28.102 INFO:teuthology.orchestra.run.smithi067.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-12-09T20:32:28.102 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:28.102 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:32:28.103 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:28.103 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package (+2 Dependent packages) 2019-12-09T20:32:28.103 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:28.103 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 316 k 2019-12-09T20:32:28.103 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.1 M 2019-12-09T20:32:28.104 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:32:28.639 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:28.640 INFO:teuthology.orchestra.run.smithi067.stdout:Total 587 kB/s | 316 kB 00:00 2019-12-09T20:32:28.643 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:32:28.652 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:32:28.732 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:32:28.732 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:32:28.788 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:28.788 INFO:teuthology.orchestra.run.smithi081.stdout:Total 371 kB/s | 316 kB 00:00 2019-12-09T20:32:28.792 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:32:28.801 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:32:28.876 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:32:28.876 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:32:29.498 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 1/3 2019-12-09T20:32:29.669 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 1/3 2019-12-09T20:32:29.786 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 2/3 2019-12-09T20:32:30.003 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 2/3 2019-12-09T20:32:30.257 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-7936.g827e03c.el7.noarch 3/3 2019-12-09T20:32:30.326 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-7936.g827e03c.el7.noarch 3/3 2019-12-09T20:32:30.369 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 1/3 2019-12-09T20:32:30.463 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 1/3 2019-12-09T20:32:30.486 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 2/3 2019-12-09T20:32:30.630 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 2/3 2019-12-09T20:32:30.963 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-7936.g827e03c.el7.noarch 3/3 2019-12-09T20:32:30.963 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:30.963 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:32:30.964 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-ssh.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:30.964 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:30.964 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:32:30.964 INFO:teuthology.orchestra.run.smithi067.stdout: python-execnet.noarch 0:1.2.0-7.el7 python-remoto.noarch 0:0.0.25-1.el7 2019-12-09T20:32:30.964 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:30.964 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:32:31.064 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-7936.g827e03c.el7.noarch 3/3 2019-12-09T20:32:31.065 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:31.065 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:32:31.065 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-mgr-ssh.noarch 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:31.065 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:31.065 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Installed: 2019-12-09T20:32:31.065 INFO:teuthology.orchestra.run.smithi081.stdout: python-execnet.noarch 0:1.2.0-7.el7 python-remoto.noarch 0:0.0.25-1.el7 2019-12-09T20:32:31.065 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:31.066 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:32:31.089 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2019-12-09T20:32:31.213 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install ceph-fuse 2019-12-09T20:32:31.304 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:31.344 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:31.345 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:31.345 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:31.345 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:31.346 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:31.430 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:31.472 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:31.473 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:31.473 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:31.474 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:31.474 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:32.001 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:32.125 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:32.342 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:32:32.343 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:32.343 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:32.354 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:32:32.388 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python3 for package: 2:ceph-fuse-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:32:32.388 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:32.389 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-12-09T20:32:32.389 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python3-libs(x86-64) = 3.6.8-10.el7 for package: python3-3.6.8-10.el7.x86_64 2019-12-09T20:32:32.390 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-12-09T20:32:32.391 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-12-09T20:32:32.393 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-10.el7.x86_64 2019-12-09T20:32:32.393 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:32.393 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-12-09T20:32:32.399 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-12-09T20:32:32.399 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-12-09T20:32:32.455 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:32:32.455 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:32.455 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:32.466 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:32:32.498 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python3 for package: 2:ceph-fuse-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:32:32.499 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:32.499 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-12-09T20:32:32.499 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python3-libs(x86-64) = 3.6.8-10.el7 for package: python3-3.6.8-10.el7.x86_64 2019-12-09T20:32:32.501 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-12-09T20:32:32.502 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-12-09T20:32:32.503 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python3-3.6.8-10.el7.x86_64 2019-12-09T20:32:32.503 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:32.503 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-12-09T20:32:32.509 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-12-09T20:32:32.509 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-12-09T20:32:32.780 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:32.875 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:33.012 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:33.012 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:32:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:33.013 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository 2019-12-09T20:32:33.013 INFO:teuthology.orchestra.run.smithi067.stdout: Size 2019-12-09T20:32:33.013 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:33.013 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:32:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:15.0.0-7936.g827e03c.el7 ceph 566 k 2019-12-09T20:32:33.014 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:32:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3 x86_64 3.6.8-10.el7 base 69 k 2019-12-09T20:32:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-libs x86_64 3.6.8-10.el7 base 7.0 M 2019-12-09T20:32:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pip noarch 9.0.3-5.el7 base 1.8 M 2019-12-09T20:32:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: python3-setuptools noarch 39.2.0-10.el7 base 629 k 2019-12-09T20:32:33.014 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:33.015 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:32:33.015 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:33.015 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package (+4 Dependent packages) 2019-12-09T20:32:33.015 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:33.016 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 10 M 2019-12-09T20:32:33.016 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 49 M 2019-12-09T20:32:33.016 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:32:33.097 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:33.097 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:32:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:33.098 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository 2019-12-09T20:32:33.098 INFO:teuthology.orchestra.run.smithi081.stdout: Size 2019-12-09T20:32:33.098 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:33.098 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:32:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse x86_64 2:15.0.0-7936.g827e03c.el7 ceph 566 k 2019-12-09T20:32:33.099 INFO:teuthology.orchestra.run.smithi081.stdout:Installing for dependencies: 2019-12-09T20:32:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: python3 x86_64 3.6.8-10.el7 base 69 k 2019-12-09T20:32:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: python3-libs x86_64 3.6.8-10.el7 base 7.0 M 2019-12-09T20:32:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pip noarch 9.0.3-5.el7 base 1.8 M 2019-12-09T20:32:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: python3-setuptools noarch 39.2.0-10.el7 base 629 k 2019-12-09T20:32:33.099 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:33.100 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:32:33.100 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:33.100 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package (+4 Dependent packages) 2019-12-09T20:32:33.100 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:33.101 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 10 M 2019-12-09T20:32:33.101 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 49 M 2019-12-09T20:32:33.101 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:32:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:Total 21 MB/s | 10 MB 00:00 2019-12-09T20:32:33.487 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:32:33.502 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:32:33.604 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:33.605 INFO:teuthology.orchestra.run.smithi081.stdout:Total 20 MB/s | 10 MB 00:00 2019-12-09T20:32:33.620 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:32:33.621 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:32:33.622 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:32:33.643 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:32:33.768 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:32:33.769 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:32:35.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 1/5 2019-12-09T20:32:36.149 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 1/5 2019-12-09T20:32:36.450 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-3.6.8-10.el7.x86_64 2/5 2019-12-09T20:32:36.704 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-3.6.8-10.el7.x86_64 2/5 2019-12-09T20:32:37.337 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 3/5 2019-12-09T20:32:37.634 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 3/5 2019-12-09T20:32:37.795 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 4/5 2019-12-09T20:32:38.064 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:ceph-fuse-15.0.0-7936.g827e03c.el7.x86_64 5/5 2019-12-09T20:32:38.108 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 4/5 2019-12-09T20:32:38.171 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:ceph-fuse-15.0.0-7936.g827e03c.el7.x86_64 1/5 2019-12-09T20:32:38.280 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 2/5 2019-12-09T20:32:38.397 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 3/5 2019-12-09T20:32:38.427 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:ceph-fuse-15.0.0-7936.g827e03c.el7.x86_64 5/5 2019-12-09T20:32:38.514 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-3.6.8-10.el7.x86_64 4/5 2019-12-09T20:32:38.535 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:ceph-fuse-15.0.0-7936.g827e03c.el7.x86_64 1/5 2019-12-09T20:32:38.685 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 2/5 2019-12-09T20:32:38.811 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 3/5 2019-12-09T20:32:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 5/5 2019-12-09T20:32:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:38.949 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:32:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:38.949 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:38.949 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:32:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3.x86_64 0:3.6.8-10.el7 python3-libs.x86_64 0:3.6.8-10.el7 2019-12-09T20:32:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pip.noarch 0:9.0.3-5.el7 python3-setuptools.noarch 0:39.2.0-10.el7 2019-12-09T20:32:38.950 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:38.950 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:32:38.969 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-3.6.8-10.el7.x86_64 4/5 2019-12-09T20:32:39.076 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2019-12-09T20:32:39.308 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:39.350 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:39.351 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:39.351 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:39.351 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:39.352 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:39.513 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 5/5 2019-12-09T20:32:39.513 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:39.513 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:32:39.513 INFO:teuthology.orchestra.run.smithi081.stdout: ceph-fuse.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:39.514 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:39.514 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Installed: 2019-12-09T20:32:39.514 INFO:teuthology.orchestra.run.smithi081.stdout: python3.x86_64 0:3.6.8-10.el7 python3-libs.x86_64 0:3.6.8-10.el7 2019-12-09T20:32:39.514 INFO:teuthology.orchestra.run.smithi081.stdout: python3-pip.noarch 0:9.0.3-5.el7 python3-setuptools.noarch 0:39.2.0-10.el7 2019-12-09T20:32:39.515 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:39.515 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:32:39.630 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs2 2019-12-09T20:32:39.862 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:39.906 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:39.907 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:39.907 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:39.908 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:39.908 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:40.017 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:40.364 INFO:teuthology.orchestra.run.smithi067.stdout:Package 2:libcephfs2-15.0.0-7936.g827e03c.el7.x86_64 already installed and latest version 2019-12-09T20:32:40.365 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do 2019-12-09T20:32:40.426 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2019-12-09T20:32:40.552 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:40.643 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:40.688 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:40.689 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:40.690 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:40.885 INFO:teuthology.orchestra.run.smithi081.stdout:Package 2:libcephfs2-15.0.0-7936.g827e03c.el7.x86_64 already installed and latest version 2019-12-09T20:32:40.885 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do 2019-12-09T20:32:40.944 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install libcephfs-devel 2019-12-09T20:32:41.171 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:41.212 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:41.213 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:41.214 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:41.214 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:41.215 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:41.357 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:41.704 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:32:41.704 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:41.704 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-7936.g827e03c.el7 for package: 2:libcephfs-devel-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:32:41.737 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:41.737 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package librados-devel.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:41.860 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:42.129 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:42.187 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:32:42.187 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:42.187 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:42.188 INFO:teuthology.orchestra.run.smithi081.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-7936.g827e03c.el7 for package: 2:libcephfs-devel-15.0.0-7936.g827e03c.el7.x86_64 2019-12-09T20:32:42.218 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:42.218 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package librados-devel.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:42.364 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:42.364 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:32:42.365 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:42.365 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:42.365 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository 2019-12-09T20:32:42.365 INFO:teuthology.orchestra.run.smithi067.stdout: Size 2019-12-09T20:32:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:32:42.366 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:15.0.0-7936.g827e03c.el7 ceph 17 k 2019-12-09T20:32:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:32:42.366 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:15.0.0-7936.g827e03c.el7 ceph 153 k 2019-12-09T20:32:42.366 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:42.366 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:32:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package (+1 Dependent package) 2019-12-09T20:32:42.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 170 k 2019-12-09T20:32:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 679 k 2019-12-09T20:32:42.367 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:32:42.590 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:42.814 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:42.815 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:32:42.816 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:42.816 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:42.816 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository 2019-12-09T20:32:42.816 INFO:teuthology.orchestra.run.smithi081.stdout: Size 2019-12-09T20:32:42.816 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:42.816 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:32:42.817 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel x86_64 2:15.0.0-7936.g827e03c.el7 ceph 17 k 2019-12-09T20:32:42.817 INFO:teuthology.orchestra.run.smithi081.stdout:Installing for dependencies: 2019-12-09T20:32:42.817 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel x86_64 2:15.0.0-7936.g827e03c.el7 ceph 153 k 2019-12-09T20:32:42.817 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:42.817 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:32:42.818 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:42.818 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package (+1 Dependent package) 2019-12-09T20:32:42.818 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:42.818 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 170 k 2019-12-09T20:32:42.818 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 679 k 2019-12-09T20:32:42.819 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:32:42.946 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:42.946 INFO:teuthology.orchestra.run.smithi067.stdout:Total 293 kB/s | 170 kB 00:00 2019-12-09T20:32:42.949 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:32:42.961 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:32:42.964 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:32:42.964 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:32:43.503 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:librados-devel-15.0.0-7936.g827e03c.el7.x86_64 1/2 2019-12-09T20:32:43.640 INFO:teuthology.orchestra.run.smithi081.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:32:43.640 INFO:teuthology.orchestra.run.smithi081.stdout:Total 206 kB/s | 170 kB 00:00 2019-12-09T20:32:43.643 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:32:43.656 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:32:43.660 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:32:43.660 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:32:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:libcephfs-devel-15.0.0-7936.g827e03c.el7.x86_64 2/2 2019-12-09T20:32:43.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:libcephfs-devel-15.0.0-7936.g827e03c.el7.x86_64 1/2 2019-12-09T20:32:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:librados-devel-15.0.0-7936.g827e03c.el7.x86_64 2/2 2019-12-09T20:32:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:44.396 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:32:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:44.396 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:44.396 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:32:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:44.397 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:44.397 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:32:44.520 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2019-12-09T20:32:44.720 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:44.726 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:librados-devel-15.0.0-7936.g827e03c.el7.x86_64 1/2 2019-12-09T20:32:44.759 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:44.759 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:44.760 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:44.761 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:45.022 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:libcephfs-devel-15.0.0-7936.g827e03c.el7.x86_64 2/2 2019-12-09T20:32:45.169 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:libcephfs-devel-15.0.0-7936.g827e03c.el7.x86_64 1/2 2019-12-09T20:32:45.424 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:45.687 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:librados-devel-15.0.0-7936.g827e03c.el7.x86_64 2/2 2019-12-09T20:32:45.687 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:45.688 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:32:45.688 INFO:teuthology.orchestra.run.smithi081.stdout: libcephfs-devel.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:45.688 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:45.688 INFO:teuthology.orchestra.run.smithi081.stdout:Dependency Installed: 2019-12-09T20:32:45.688 INFO:teuthology.orchestra.run.smithi081.stdout: librados-devel.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:45.688 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:45.688 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:32:45.765 INFO:teuthology.orchestra.run.smithi067.stdout:Package 2:librados2-15.0.0-7936.g827e03c.el7.x86_64 already installed and latest version 2019-12-09T20:32:45.765 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do 2019-12-09T20:32:45.811 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install librados2 2019-12-09T20:32:45.827 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2019-12-09T20:32:45.996 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:46.032 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:46.033 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:46.034 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:46.034 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:46.034 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:46.070 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:46.115 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:46.116 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:46.116 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:46.117 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:46.117 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:46.658 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:46.772 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:47.001 INFO:teuthology.orchestra.run.smithi067.stdout:Package 2:librbd1-15.0.0-7936.g827e03c.el7.x86_64 already installed and latest version 2019-12-09T20:32:47.001 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do 2019-12-09T20:32:47.063 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install python-ceph 2019-12-09T20:32:47.107 INFO:teuthology.orchestra.run.smithi081.stdout:Package 2:librados2-15.0.0-7936.g827e03c.el7.x86_64 already installed and latest version 2019-12-09T20:32:47.107 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do 2019-12-09T20:32:47.166 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install librbd1 2019-12-09T20:32:47.261 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:47.299 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:47.300 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:47.300 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:47.301 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:47.369 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:47.409 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:47.410 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:47.410 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:47.410 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:47.411 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:47.938 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:48.052 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:48.278 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:32:48.278 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:48.279 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:48.376 INFO:teuthology.orchestra.run.smithi081.stdout:Package 2:librbd1-15.0.0-7936.g827e03c.el7.x86_64 already installed and latest version 2019-12-09T20:32:48.376 INFO:teuthology.orchestra.run.smithi081.stdout:Nothing to do 2019-12-09T20:32:48.436 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install python-ceph 2019-12-09T20:32:48.672 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:48.675 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:48.720 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:48.721 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:48.722 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:48.722 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:48.722 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:48.863 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:48.863 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:32:48.863 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:48.864 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:48.864 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository 2019-12-09T20:32:48.864 INFO:teuthology.orchestra.run.smithi067.stdout: Size 2019-12-09T20:32:48.864 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:48.864 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:32:48.864 INFO:teuthology.orchestra.run.smithi067.stdout: python-ceph-compat x86_64 2:15.0.0-7936.g827e03c.el7 ceph 2.2 k 2019-12-09T20:32:48.865 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:48.865 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:32:48.865 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:48.865 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2019-12-09T20:32:48.865 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:48.865 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.2 k 2019-12-09T20:32:48.865 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2019-12-09T20:32:48.866 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:32:49.166 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:32:49.167 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:32:49.169 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:32:49.170 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:32:49.366 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:49.701 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:32:49.701 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:49.701 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:49.734 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:python-ceph-compat-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:32:50.069 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:python-ceph-compat-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:32:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:50.226 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:32:50.226 INFO:teuthology.orchestra.run.smithi067.stdout: python-ceph-compat.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:50.227 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:50.227 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:32:50.252 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:50.252 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:32:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:50.253 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:50.253 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository 2019-12-09T20:32:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: Size 2019-12-09T20:32:50.254 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:50.254 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:32:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: python-ceph-compat x86_64 2:15.0.0-7936.g827e03c.el7 ceph 2.2 k 2019-12-09T20:32:50.254 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:50.254 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:32:50.254 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:50.255 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2019-12-09T20:32:50.255 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:50.255 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 2.2 k 2019-12-09T20:32:50.255 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 0 2019-12-09T20:32:50.255 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:32:50.552 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:32:50.553 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:32:50.556 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:32:50.556 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:32:50.660 INFO:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2019-12-09T20:32:50.856 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:50.895 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:50.896 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:50.896 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:50.896 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:50.897 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:51.081 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:python-ceph-compat-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:32:51.550 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:51.623 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:python-ceph-compat-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:32:51.624 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:51.624 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:32:51.624 INFO:teuthology.orchestra.run.smithi081.stdout: python-ceph-compat.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:51.624 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:51.624 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:32:51.740 INFO:teuthology.orchestra.run.smithi081:> sudo yum -y install rbd-fuse 2019-12-09T20:32:51.891 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:32:51.892 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:32:51.892 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:51.971 INFO:teuthology.orchestra.run.smithi081.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:32:52.010 INFO:teuthology.orchestra.run.smithi081.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:32:52.011 INFO:teuthology.orchestra.run.smithi081.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:32:52.012 INFO:teuthology.orchestra.run.smithi081.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:32:52.012 INFO:teuthology.orchestra.run.smithi081.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:32:52.012 INFO:teuthology.orchestra.run.smithi081.stdout: * updates: mirror.jaleco.com 2019-12-09T20:32:52.290 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:52.481 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:52.481 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:32:52.482 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:52.482 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:52.482 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2019-12-09T20:32:52.483 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:52.483 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:32:52.483 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:15.0.0-7936.g827e03c.el7 ceph 76 k 2019-12-09T20:32:52.483 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:52.483 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:32:52.483 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:32:52.483 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2019-12-09T20:32:52.484 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:52.484 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 76 k 2019-12-09T20:32:52.484 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 200 k 2019-12-09T20:32:52.484 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:32:52.654 INFO:teuthology.orchestra.run.smithi081.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:32:52.827 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:32:52.850 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:32:52.851 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:32:52.852 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:32:52.980 INFO:teuthology.orchestra.run.smithi081.stdout:Resolving Dependencies 2019-12-09T20:32:52.980 INFO:teuthology.orchestra.run.smithi081.stdout:--> Running transaction check 2019-12-09T20:32:52.980 INFO:teuthology.orchestra.run.smithi081.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-7936.g827e03c.el7 will be installed 2019-12-09T20:32:53.353 INFO:teuthology.orchestra.run.smithi081.stdout:--> Finished Dependency Resolution 2019-12-09T20:32:53.471 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : 2:rbd-fuse-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:32:53.535 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:53.535 INFO:teuthology.orchestra.run.smithi081.stdout:Dependencies Resolved 2019-12-09T20:32:53.536 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:53.536 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:53.536 INFO:teuthology.orchestra.run.smithi081.stdout: Package Arch Version Repository Size 2019-12-09T20:32:53.537 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:53.537 INFO:teuthology.orchestra.run.smithi081.stdout:Installing: 2019-12-09T20:32:53.537 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse x86_64 2:15.0.0-7936.g827e03c.el7 ceph 76 k 2019-12-09T20:32:53.537 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:53.537 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction Summary 2019-12-09T20:32:53.537 INFO:teuthology.orchestra.run.smithi081.stdout:================================================================================ 2019-12-09T20:32:53.537 INFO:teuthology.orchestra.run.smithi081.stdout:Install 1 Package 2019-12-09T20:32:53.538 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:53.538 INFO:teuthology.orchestra.run.smithi081.stdout:Total download size: 76 k 2019-12-09T20:32:53.538 INFO:teuthology.orchestra.run.smithi081.stdout:Installed size: 200 k 2019-12-09T20:32:53.538 INFO:teuthology.orchestra.run.smithi081.stdout:Downloading packages: 2019-12-09T20:32:53.871 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : 2:rbd-fuse-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:32:53.872 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:53.872 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:32:53.872 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:53.872 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:32:53.872 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:32:53.903 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction check 2019-12-09T20:32:53.925 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction test 2019-12-09T20:32:53.927 INFO:teuthology.orchestra.run.smithi081.stdout:Transaction test succeeded 2019-12-09T20:32:53.928 INFO:teuthology.orchestra.run.smithi081.stdout:Running transaction 2019-12-09T20:32:53.989 DEBUG:teuthology.parallel:result is None 2019-12-09T20:32:54.747 INFO:teuthology.orchestra.run.smithi081.stdout: Installing : 2:rbd-fuse-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:32:55.188 INFO:teuthology.orchestra.run.smithi081.stdout: Verifying : 2:rbd-fuse-15.0.0-7936.g827e03c.el7.x86_64 1/1 2019-12-09T20:32:55.189 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:55.189 INFO:teuthology.orchestra.run.smithi081.stdout:Installed: 2019-12-09T20:32:55.189 INFO:teuthology.orchestra.run.smithi081.stdout: rbd-fuse.x86_64 2:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:55.189 INFO:teuthology.orchestra.run.smithi081.stdout: 2019-12-09T20:32:55.190 INFO:teuthology.orchestra.run.smithi081.stdout:Complete! 2019-12-09T20:32:55.305 DEBUG:teuthology.parallel:result is None 2019-12-09T20:32:55.305 INFO:teuthology.orchestra.run.smithi081:> sudo lsb_release -is 2019-12-09T20:32:55.438 INFO:teuthology.orchestra.run.smithi081.stdout:CentOS 2019-12-09T20:32:55.440 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-09T20:32:55.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=827e03c8a6b1e17d93dcaf79724437bac291c35b 2019-12-09T20:32:55.606 INFO:teuthology.orchestra.run.smithi081:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-12-09T20:32:55.724 INFO:teuthology.orchestra.run.smithi081.stdout:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:55.724 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-7936.g827e03c.el7 2019-12-09T20:32:55.724 INFO:teuthology.task.install:The correct ceph version 15.0.0-7936.g827e03c is installed. 2019-12-09T20:32:55.725 INFO:teuthology.orchestra.run.smithi067:> sudo lsb_release -is 2019-12-09T20:32:55.874 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS 2019-12-09T20:32:55.877 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-09T20:32:55.877 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=notcmalloc&distros=centos%2F7%2Fx86_64&sha1=827e03c8a6b1e17d93dcaf79724437bac291c35b 2019-12-09T20:32:56.039 INFO:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-12-09T20:32:56.178 INFO:teuthology.orchestra.run.smithi067.stdout:15.0.0-7936.g827e03c.el7 2019-12-09T20:32:56.178 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-7936.g827e03c.el7 2019-12-09T20:32:56.179 INFO:teuthology.task.install:The correct ceph version 15.0.0-7936.g827e03c is installed. 2019-12-09T20:32:56.179 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-12-09T20:32:56.179 INFO:teuthology.orchestra.run.smithi081:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-12-09T20:32:56.296 INFO:teuthology.orchestra.run.smithi067:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-12-09T20:32:56.420 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-12-09T20:32:56.421 INFO:teuthology.orchestra.run.smithi081:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-12-09T20:32:56.543 INFO:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-12-09T20:32:56.656 INFO:teuthology.orchestra.run.smithi067:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-12-09T20:32:56.779 INFO:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-12-09T20:32:56.893 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-12-09T20:32:56.893 INFO:teuthology.orchestra.run.smithi081:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-12-09T20:32:57.013 INFO:teuthology.orchestra.run.smithi081:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-12-09T20:32:57.119 INFO:teuthology.orchestra.run.smithi067:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-12-09T20:32:57.240 INFO:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-12-09T20:32:57.339 INFO:teuthology.run_tasks:Running task ceph... 2019-12-09T20:32:57.459 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2019-12-09T20:32:57.460 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2019-12-09T20:32:57.467 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 777 /var/log/ceph 2019-12-09T20:32:57.591 INFO:tasks.ceph:Disabling ceph logrotate... 2019-12-09T20:32:57.591 INFO:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2019-12-09T20:32:57.595 INFO:teuthology.orchestra.run.smithi081:> sudo rm -f -- /etc/logrotate.d/ceph 2019-12-09T20:32:57.706 INFO:tasks.ceph:Creating extra log directories... 2019-12-09T20:32:57.706 INFO:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-12-09T20:32:57.710 INFO:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-12-09T20:32:57.807 INFO:tasks.ceph:Creating ceph cluster ceph... 2019-12-09T20:32:57.807 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'global': {'osd heartbeat grace': 40, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'mon client directed command retry': 5, 'osd_min_pg_log_entries': 10}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'osd shutdown pgref assert': True, 'debug journal': 20, 'bluestore allocator': 'bitmap', 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'bdev async discard': True, 'osd fast shutdown': False, 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'client': {'debug rgw': 20, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt require ssl': False, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10}}} 2019-12-09T20:32:57.808 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'flavor': 'notcmalloc', 'sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b'}}, {'ceph': None}, {'openssl_keys': None}, {'rgw': {'client.0': {'valgrind': ['--tool=memcheck', '--max-threads=1024']}}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'rgw/test_rgw_gc_log.sh']}}}, {'ragweed': {'client.0': {'rgw_server': 'client.0', 'stages': 'prepare,check', 'default-branch': 'ceph-master'}}}, {'s3tests-java': {'client.0': {'force-repo': 'https://github.com/ceph/java_s3tests.git', 'force-branch': 'master'}}}, {'s3tests': {'client.0': {'rgw_server': 'client.0', 'force-branch': 'public-access-qa', 'git_remote': 'git://github.com/theanalyst/'}}}, {'swift': {'client.0': {'rgw_server': 'client.0'}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-04-1433/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.13068', 'owner': 'scheduled_abhi@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'targets': {'smithi067.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2H9fW2R5/trFss4jyqgkL9w8xuX6elX8nGrQ+RL8mge/gi7oAgEIdS8nZZR6mOIMHJtAkRkbGdZRDeNedeUF+/vGVyLWDV6gjYBNid7Sm3NnLWhay2O5vtqilGTd9xa2RDN3KDkv3br154iaTnZGYhdvqZbdb3v8oUhnh5mto4vAvS2irdzADKDX81W83PM4uPuOB8VrWIkYxx795HAV0TUb+e3jdpNx0wnfR4aw+0BzBOgYLUhv8bDX3BwZSRJu8155N9evjOUg+K/G4hkjqXb+FUN9mmi32d4kKbGzTmX9awvr4p/iKR71hkRbXRWGBBJlakLjAAXphx7dxpy3b', 'smithi081.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyft3l0JDlpGYgd1tPqZYy5sl97AbJOYYuBJUczv3OJJ+5Mjm9vmrNS3su0hKng/jm+JsyqGwS7WkjupHLH8StHd9CVBY7N6m+SLs+MR5mVaaOA8gg/BtaCZKkAT9CVNd4+z6wV7SBqawNfJnun5zEFZlgmadUtirejdtvvDOroc7MvW8SBXLm+AP5m8Kq5tjPq/nxA+O47yvCSuteI1XynmIC++xxGjbt2K/HBm8KA9Xqq6IA8Fy732r8NI3C/syMeMlSGHu7jrZTC8arYcfcspKwOd1mC4bfqviL7bP30BTxRoWCfE+NvNdcFFvdXVoHc5mmirjS1XTZmh9Zalf3'}, 'job_id': '4585197', 'tube': 'smithi', 'suite_branch': 'wip-abhi-testing-2019-12-04-1433', 'priority': 99, 'first_in_suite': False, 'branch': 'wip-abhi-testing-2019-12-04-1433', 'suite': 'rgw', 'email': None, 'archive_path': '/home/teuthworker/archive/abhi-2019-12-09_15:07:12-rgw-wip-abhi-testing-2019-12-04-1433-distro-basic-smithi/4585197', 'description': 'rgw/verify/{clusters/fixed-2.yaml frontend/civetweb.yaml msgr-failures/few.yaml objectstore/bluestore-bitmap.yaml overrides.yaml proto/http.yaml rgw_pool_type/ec-profile.yaml tasks/{0-install.yaml cls.yaml ragweed.yaml s3tests-java.yaml s3tests.yaml swift.yaml} validater/valgrind.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b', 'name': 'abhi-2019-12-09_15:07:12-rgw-wip-abhi-testing-2019-12-04-1433-distro-basic-smithi', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b', 'os_type': 'centos', 'overrides': {'s3tests': {'slow_backend': True}, 'ceph-deploy': {'fs': 'xfs', 'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'mon osd full ratio': 0.9, 'mon osd backfillfull_ratio': 0.85, 'bluestore fsck on mount': True, 'mon osd nearfull ratio': 0.8, 'debug bluestore': 20, 'debug bluefs': 20, 'osd objectstore': 'bluestore', 'bluestore block size': 96636764160, 'debug rocksdb': 10, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95, 'bdev async discard': True}}}, 'rgw': {'frontend': 'civetweb', 'compression type': 'random', 'storage classes': 'LUKEWARM, FROZEN', 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 3, 'm': 1, 'name': 'testprofile'}, 'ec-data-pool': True}, 'workunit': {'sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b', 'branch': 'wip-abhi-testing-2019-12-04-1433'}, 'ceph': {'conf': {'global': {'osd heartbeat grace': 40, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'mon client directed command retry': 5, 'osd_min_pg_log_entries': 10}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'osd shutdown pgref assert': True, 'debug journal': 20, 'bluestore allocator': 'bitmap', 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'bdev async discard': True, 'osd fast shutdown': False, 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon osd crush smoke test': False, 'debug ms': 1}, 'client': {'debug rgw': 20, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt require ssl': False, 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw lc debug interval': 10}}, 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}, 'fs': 'xfs', 'sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b'}, 'install': {'ceph': {'flavor': 'notcmalloc', 'sha1': '827e03c8a6b1e17d93dcaf79724437bac291c35b'}}, 'admin_socket': {'branch': 'wip-abhi-testing-2019-12-04-1433'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}}} 2019-12-09T20:32:57.809 INFO:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-12-09T20:32:57.813 INFO:teuthology.orchestra.run.smithi081:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-12-09T20:32:57.887 INFO:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2019-12-09T20:32:57.890 INFO:teuthology.orchestra.run.smithi081:> sudo install -d -m0777 -- /var/run/ceph 2019-12-09T20:32:58.095 DEBUG:teuthology.orchestra.remote:smithi081:/scratch_devs is 72B 2019-12-09T20:32:58.172 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-09T20:32:58.173 INFO:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-12-09T20:32:58.239 INFO:teuthology.orchestra.run.smithi081.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2019-12-09T20:32:58.239 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-09T20:32:58.239 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 100104 Links: 1 2019-12-09T20:32:58.240 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-09T20:32:58.240 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2019-12-09T20:32:58.240 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2019-12-09 20:31:44.618298788 +0000 2019-12-09T20:32:58.241 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2019-12-09 20:29:51.201734720 +0000 2019-12-09T20:32:58.241 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2019-12-09 20:29:51.201734720 +0000 2019-12-09T20:32:58.241 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2019-12-09T20:32:58.262 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2019-12-09T20:32:58.263 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2019-12-09T20:32:58.263 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes (512 B) copied, 0.000344992 s, 1.5 MB/s 2019-12-09T20:32:58.267 INFO:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-12-09T20:32:58.329 INFO:teuthology.orchestra.run.smithi081.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2019-12-09T20:32:58.329 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-09T20:32:58.330 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 100141 Links: 1 2019-12-09T20:32:58.330 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-09T20:32:58.330 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2019-12-09T20:32:58.330 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2019-12-09 20:31:44.618298788 +0000 2019-12-09T20:32:58.330 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2019-12-09 20:29:51.669724106 +0000 2019-12-09T20:32:58.330 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2019-12-09 20:29:51.669724106 +0000 2019-12-09T20:32:58.330 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2019-12-09T20:32:58.351 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2019-12-09T20:32:58.351 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2019-12-09T20:32:58.351 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes (512 B) copied, 0.000346549 s, 1.5 MB/s 2019-12-09T20:32:58.356 INFO:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-12-09T20:32:58.420 INFO:teuthology.orchestra.run.smithi081.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2019-12-09T20:32:58.420 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-09T20:32:58.420 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 92978 Links: 1 2019-12-09T20:32:58.420 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-09T20:32:58.421 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2019-12-09T20:32:58.421 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2019-12-09 20:31:44.618298788 +0000 2019-12-09T20:32:58.421 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2019-12-09 20:29:52.146713287 +0000 2019-12-09T20:32:58.421 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2019-12-09 20:29:52.146713287 +0000 2019-12-09T20:32:58.421 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2019-12-09T20:32:58.444 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2019-12-09T20:32:58.444 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2019-12-09T20:32:58.444 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes (512 B) copied, 0.000343261 s, 1.5 MB/s 2019-12-09T20:32:58.449 INFO:teuthology.orchestra.run.smithi081:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-12-09T20:32:58.505 INFO:teuthology.orchestra.run.smithi081.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2019-12-09T20:32:58.505 INFO:teuthology.orchestra.run.smithi081.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-09T20:32:58.505 INFO:teuthology.orchestra.run.smithi081.stdout:Device: 5h/5d Inode: 100204 Links: 1 2019-12-09T20:32:58.506 INFO:teuthology.orchestra.run.smithi081.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-09T20:32:58.506 INFO:teuthology.orchestra.run.smithi081.stdout:Context: system_u:object_r:device_t:s0 2019-12-09T20:32:58.506 INFO:teuthology.orchestra.run.smithi081.stdout:Access: 2019-12-09 20:31:44.618298788 +0000 2019-12-09T20:32:58.506 INFO:teuthology.orchestra.run.smithi081.stdout:Modify: 2019-12-09 20:29:52.610703152 +0000 2019-12-09T20:32:58.506 INFO:teuthology.orchestra.run.smithi081.stdout:Change: 2019-12-09 20:29:52.610703152 +0000 2019-12-09T20:32:58.507 INFO:teuthology.orchestra.run.smithi081.stdout: Birth: - 2019-12-09T20:32:58.525 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records in 2019-12-09T20:32:58.526 INFO:teuthology.orchestra.run.smithi081.stderr:1+0 records out 2019-12-09T20:32:58.526 INFO:teuthology.orchestra.run.smithi081.stderr:512 bytes (512 B) copied, 0.00032673 s, 1.6 MB/s 2019-12-09T20:32:58.530 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-12-09T20:32:58.530 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-09T20:32:58.530 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-12-09T20:32:58.530 INFO:tasks.ceph:dev map: {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-09T20:32:58.629 DEBUG:teuthology.orchestra.remote:smithi067:/scratch_devs is 72B 2019-12-09T20:32:58.721 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-09T20:32:58.722 INFO:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-12-09T20:32:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2019-12-09T20:32:58.804 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-09T20:32:58.804 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 106288 Links: 1 2019-12-09T20:32:58.804 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-09T20:32:58.805 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2019-12-09T20:32:58.805 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2019-12-09 20:31:45.783928512 +0000 2019-12-09T20:32:58.805 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2019-12-09 20:29:52.049399243 +0000 2019-12-09T20:32:58.805 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2019-12-09 20:29:52.049399243 +0000 2019-12-09T20:32:58.805 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2019-12-09T20:32:58.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2019-12-09T20:32:58.833 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2019-12-09T20:32:58.833 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes (512 B) copied, 0.000465522 s, 1.1 MB/s 2019-12-09T20:32:58.837 INFO:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-12-09T20:32:58.912 INFO:teuthology.orchestra.run.smithi067.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2019-12-09T20:32:58.912 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-09T20:32:58.913 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 97046 Links: 1 2019-12-09T20:32:58.913 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-09T20:32:58.913 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2019-12-09T20:32:58.913 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2019-12-09 20:31:45.783928512 +0000 2019-12-09T20:32:58.913 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2019-12-09 20:29:52.528388299 +0000 2019-12-09T20:32:58.913 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2019-12-09 20:29:52.528388299 +0000 2019-12-09T20:32:58.913 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2019-12-09T20:32:58.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2019-12-09T20:32:58.937 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2019-12-09T20:32:58.937 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes (512 B) copied, 0.000421338 s, 1.2 MB/s 2019-12-09T20:32:58.942 INFO:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-12-09T20:32:59.009 INFO:teuthology.orchestra.run.smithi067.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2019-12-09T20:32:59.009 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-09T20:32:59.009 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 97077 Links: 1 2019-12-09T20:32:59.009 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-09T20:32:59.009 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2019-12-09T20:32:59.009 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2019-12-09 20:31:45.783928512 +0000 2019-12-09T20:32:59.010 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2019-12-09 20:29:52.988377790 +0000 2019-12-09T20:32:59.010 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2019-12-09 20:29:52.988377790 +0000 2019-12-09T20:32:59.010 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2019-12-09T20:32:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2019-12-09T20:32:59.032 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2019-12-09T20:32:59.033 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes (512 B) copied, 0.000350314 s, 1.5 MB/s 2019-12-09T20:32:59.037 INFO:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-12-09T20:32:59.099 INFO:teuthology.orchestra.run.smithi067.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2019-12-09T20:32:59.100 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-09T20:32:59.100 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 101902 Links: 1 2019-12-09T20:32:59.100 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-09T20:32:59.100 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2019-12-09T20:32:59.100 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2019-12-09 20:31:45.783928512 +0000 2019-12-09T20:32:59.100 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2019-12-09 20:29:53.463366938 +0000 2019-12-09T20:32:59.101 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2019-12-09 20:29:53.463366938 +0000 2019-12-09T20:32:59.101 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2019-12-09T20:32:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2019-12-09T20:32:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2019-12-09T20:32:59.120 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes (512 B) copied, 0.000406542 s, 1.3 MB/s 2019-12-09T20:32:59.123 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-12-09T20:32:59.123 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-09T20:32:59.124 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-12-09T20:32:59.124 INFO:tasks.ceph:dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-09T20:32:59.124 INFO:tasks.ceph:Generating config... 2019-12-09T20:32:59.125 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2019-12-09T20:32:59.126 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2019-12-09T20:32:59.126 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 10 2019-12-09T20:32:59.126 INFO:tasks.ceph:[global] mon client directed command retry = 5 2019-12-09T20:32:59.126 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 10 2019-12-09T20:32:59.126 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2019-12-09T20:32:59.126 INFO:tasks.ceph:[osd] debug ms = 1 2019-12-09T20:32:59.127 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2019-12-09T20:32:59.127 INFO:tasks.ceph:[osd] debug osd = 25 2019-12-09T20:32:59.127 INFO:tasks.ceph:[osd] debug bluestore = 20 2019-12-09T20:32:59.127 INFO:tasks.ceph:[osd] debug bluefs = 20 2019-12-09T20:32:59.127 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2019-12-09T20:32:59.127 INFO:tasks.ceph:[osd] debug rocksdb = 10 2019-12-09T20:32:59.127 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2019-12-09T20:32:59.127 INFO:tasks.ceph:[osd] debug journal = 20 2019-12-09T20:32:59.128 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2019-12-09T20:32:59.128 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2019-12-09T20:32:59.130 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2019-12-09T20:32:59.130 INFO:tasks.ceph:[osd] bdev async discard = True 2019-12-09T20:32:59.130 INFO:tasks.ceph:[osd] osd fast shutdown = False 2019-12-09T20:32:59.130 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2019-12-09T20:32:59.130 INFO:tasks.ceph:[osd] debug filestore = 20 2019-12-09T20:32:59.131 INFO:tasks.ceph:[osd] bdev enable discard = True 2019-12-09T20:32:59.131 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2019-12-09T20:32:59.131 INFO:tasks.ceph:[mon] debug mon = 20 2019-12-09T20:32:59.131 INFO:tasks.ceph:[mon] debug paxos = 20 2019-12-09T20:32:59.132 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2019-12-09T20:32:59.132 INFO:tasks.ceph:[mon] debug ms = 1 2019-12-09T20:32:59.132 INFO:tasks.ceph:[client] debug rgw = 20 2019-12-09T20:32:59.132 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2019-12-09T20:32:59.132 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2019-12-09T20:32:59.132 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2019-12-09T20:32:59.133 INFO:tasks.ceph:[client] rgw lc debug interval = 10 2019-12-09T20:32:59.133 INFO:tasks.ceph:Setting up mon.a... 2019-12-09T20:32:59.133 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2019-12-09T20:32:59.263 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2019-12-09T20:32:59.268 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2019-12-09T20:32:59.395 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-09T20:32:59.499 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.c', '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]'), ('mon.b', '172.21.15.81')] 2019-12-09T20:32:59.499 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.67 --addv c '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]' --add b 172.21.15.81 --print /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:32:59.647 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:32:59.648 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid bb0ccc8e-29f2-48f9-838e-c4ccdeefe22d 2019-12-09T20:32:59.648 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2019-12-09T20:32:59.648 INFO:teuthology.orchestra.run.smithi067.stdout:fsid bb0ccc8e-29f2-48f9-838e-c4ccdeefe22d 2019-12-09T20:32:59.648 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2019-12-09T20:32:59.635576+0000 2019-12-09T20:32:59.648 INFO:teuthology.orchestra.run.smithi067.stdout:created 2019-12-09T20:32:59.635576+0000 2019-12-09T20:32:59.648 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 0 (unknown) 2019-12-09T20:32:59.648 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2019-12-09T20:32:59.649 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] mon.b 2019-12-09T20:32:59.649 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2019-12-09T20:32:59.649 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2019-12-09T20:32:59.650 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID bb0ccc8e-29f2-48f9-838e-c4ccdeefe22d... 2019-12-09T20:32:59.651 INFO:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-12-09T20:32:59.655 INFO:teuthology.orchestra.run.smithi081:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-12-09T20:32:59.882 INFO:tasks.ceph:Creating admin key on mon.a... 2019-12-09T20:32:59.900 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2019-12-09T20:33:00.079 INFO:tasks.ceph:Copying monmap to all nodes... 2019-12-09T20:33:00.175 DEBUG:teuthology.orchestra.remote:smithi067:/etc/ceph/ceph.keyring is 206B 2019-12-09T20:33:00.360 DEBUG:teuthology.orchestra.remote:smithi067:/home/ubuntu/cephtest/ceph.monmap is 379B 2019-12-09T20:33:00.449 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:33:00.449 INFO:teuthology.orchestra.run.smithi081:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-09T20:33:00.600 INFO:teuthology.orchestra.run.smithi081:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:00.678 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:33:00.678 INFO:teuthology.orchestra.run.smithi067:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-09T20:33:00.831 INFO:teuthology.orchestra.run.smithi067:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:00.906 INFO:tasks.ceph:Setting up mon nodes... 2019-12-09T20:33:00.907 INFO:tasks.ceph:Setting up mgr nodes... 2019-12-09T20:33:00.907 INFO:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2019-12-09T20:33:01.102 INFO:teuthology.orchestra.run.smithi081.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2019-12-09T20:33:01.107 INFO:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2019-12-09T20:33:01.291 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2019-12-09T20:33:01.296 INFO:tasks.ceph:Setting up mds nodes... 2019-12-09T20:33:01.296 INFO:tasks.ceph_client:Setting up client nodes... 2019-12-09T20:33:01.296 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2019-12-09T20:33:01.446 INFO:teuthology.orchestra.run.smithi081.stdout:creating /etc/ceph/ceph.client.1.keyring 2019-12-09T20:33:01.484 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2019-12-09T20:33:01.639 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2019-12-09T20:33:01.673 INFO:tasks.ceph:Running mkfs on osd nodes... 2019-12-09T20:33:01.674 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi081.front.sepia.ceph.com'): {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'}} 2019-12-09T20:33:01.674 INFO:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2019-12-09T20:33:01.794 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-09T20:33:01.795 INFO:tasks.ceph:{} 2019-12-09T20:33:01.795 INFO:tasks.ceph:osd.4 2019-12-09T20:33:01.795 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:33:01.795 INFO:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-12-09T20:33:02.373 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-12-09T20:33:02.373 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-09T20:33:02.373 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=0, sparse=0 2019-12-09T20:33:02.373 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-09T20:33:02.374 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2019-12-09T20:33:02.374 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-09T20:33:02.374 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-09T20:33:02.374 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-09T20:33:02.375 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-09T20:33:02.376 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2019-12-09T20:33:02.376 INFO:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4 2019-12-09T20:33:02.564 INFO:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2019-12-09T20:33:02.689 INFO:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2019-12-09T20:33:02.792 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-09T20:33:02.792 INFO:tasks.ceph:{} 2019-12-09T20:33:02.793 INFO:tasks.ceph:osd.5 2019-12-09T20:33:02.793 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:33:02.793 INFO:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-12-09T20:33:03.393 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-12-09T20:33:03.393 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-09T20:33:03.393 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=0, sparse=0 2019-12-09T20:33:03.394 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-09T20:33:03.394 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2019-12-09T20:33:03.394 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-09T20:33:03.394 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-09T20:33:03.394 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-09T20:33:03.394 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-09T20:33:03.396 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2019-12-09T20:33:03.396 INFO:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-5 2019-12-09T20:33:03.553 INFO:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2019-12-09T20:33:03.655 INFO:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2019-12-09T20:33:03.752 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-09T20:33:03.752 INFO:tasks.ceph:{} 2019-12-09T20:33:03.752 INFO:tasks.ceph:osd.6 2019-12-09T20:33:03.753 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:33:03.753 INFO:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-12-09T20:33:04.324 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-12-09T20:33:04.324 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-09T20:33:04.324 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=0, sparse=0 2019-12-09T20:33:04.325 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-09T20:33:04.325 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2019-12-09T20:33:04.325 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-09T20:33:04.325 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-09T20:33:04.326 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-09T20:33:04.326 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-09T20:33:04.327 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2019-12-09T20:33:04.328 INFO:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2019-12-09T20:33:04.492 INFO:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2019-12-09T20:33:04.582 INFO:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2019-12-09T20:33:04.659 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-09T20:33:04.659 INFO:tasks.ceph:{} 2019-12-09T20:33:04.659 INFO:tasks.ceph:osd.7 2019-12-09T20:33:04.659 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:33:04.660 INFO:teuthology.orchestra.run.smithi081:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-12-09T20:33:05.203 INFO:teuthology.orchestra.run.smithi081.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-12-09T20:33:05.203 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-09T20:33:05.203 INFO:teuthology.orchestra.run.smithi081.stdout: = crc=1 finobt=0, sparse=0 2019-12-09T20:33:05.204 INFO:teuthology.orchestra.run.smithi081.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-09T20:33:05.204 INFO:teuthology.orchestra.run.smithi081.stdout: = sunit=0 swidth=0 blks 2019-12-09T20:33:05.204 INFO:teuthology.orchestra.run.smithi081.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-09T20:33:05.204 INFO:teuthology.orchestra.run.smithi081.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-09T20:33:05.204 INFO:teuthology.orchestra.run.smithi081.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-09T20:33:05.204 INFO:teuthology.orchestra.run.smithi081.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-09T20:33:05.205 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi081.front.sepia.ceph.com -o noatime 2019-12-09T20:33:05.205 INFO:teuthology.orchestra.run.smithi081:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-7 2019-12-09T20:33:05.360 INFO:teuthology.orchestra.run.smithi081:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2019-12-09T20:33:05.467 INFO:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:05.608 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:05.609+0000 7f5e958d5b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2019-12-09T20:33:05.610 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:05.612+0000 7f5e958d5b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2019-12-09T20:33:05.611 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:05.612+0000 7f5e958d5b40 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-09T20:33:05.611 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:05.612+0000 7f5e958d5b40 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-09T20:33:05.611 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:05.612+0000 7f5e958d5b40 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-09T20:33:05.612 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:05.614+0000 7f5e958d5b40 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2019-12-09T20:33:07.924 INFO:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2019-12-09T20:33:08.045 INFO:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:08.165 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:08.166+0000 7fc44b8f4b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2019-12-09T20:33:08.166 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:08.167+0000 7fc44b8f4b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2019-12-09T20:33:08.166 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:08.167+0000 7fc44b8f4b40 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-09T20:33:08.166 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:08.167+0000 7fc44b8f4b40 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-09T20:33:08.166 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:08.167+0000 7fc44b8f4b40 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-09T20:33:08.167 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:08.168+0000 7fc44b8f4b40 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2019-12-09T20:33:10.477 INFO:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2019-12-09T20:33:10.590 INFO:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:10.733 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:10.734+0000 7f20b9609b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2019-12-09T20:33:10.738 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:10.735+0000 7f20b9609b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2019-12-09T20:33:10.739 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:10.735+0000 7f20b9609b40 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-09T20:33:10.739 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:10.735+0000 7f20b9609b40 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-09T20:33:10.740 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:10.735+0000 7f20b9609b40 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-09T20:33:10.740 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:10.736+0000 7f20b9609b40 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2019-12-09T20:33:13.043 INFO:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2019-12-09T20:33:13.157 INFO:teuthology.orchestra.run.smithi081:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:13.313 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:13.315+0000 7f695d71cb40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2019-12-09T20:33:13.314 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:13.316+0000 7f695d71cb40 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2019-12-09T20:33:13.314 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:13.316+0000 7f695d71cb40 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-09T20:33:13.315 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:13.316+0000 7f695d71cb40 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-09T20:33:13.315 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:13.316+0000 7f695d71cb40 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-09T20:33:13.315 INFO:teuthology.orchestra.run.smithi081.stderr:2019-12-09T20:33:13.317+0000 7f695d71cb40 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2019-12-09T20:33:15.628 INFO:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2019-12-09T20:33:15.740 INFO:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2019-12-09T20:33:15.862 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-09T20:33:15.862 INFO:tasks.ceph:{} 2019-12-09T20:33:15.862 INFO:tasks.ceph:osd.0 2019-12-09T20:33:15.862 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:33:15.863 INFO:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-12-09T20:33:16.462 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-12-09T20:33:16.462 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-09T20:33:16.462 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=0, sparse=0 2019-12-09T20:33:16.462 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-09T20:33:16.462 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2019-12-09T20:33:16.462 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-09T20:33:16.463 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-09T20:33:16.463 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-09T20:33:16.463 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-09T20:33:16.464 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2019-12-09T20:33:16.464 INFO:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0 2019-12-09T20:33:16.640 INFO:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2019-12-09T20:33:16.770 INFO:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2019-12-09T20:33:16.879 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-09T20:33:16.879 INFO:tasks.ceph:{} 2019-12-09T20:33:16.880 INFO:tasks.ceph:osd.1 2019-12-09T20:33:16.880 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:33:16.880 INFO:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-12-09T20:33:17.486 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-12-09T20:33:17.486 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-09T20:33:17.486 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=0, sparse=0 2019-12-09T20:33:17.486 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-09T20:33:17.487 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2019-12-09T20:33:17.487 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-09T20:33:17.487 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-09T20:33:17.487 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-09T20:33:17.488 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-09T20:33:17.489 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2019-12-09T20:33:17.489 INFO:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2019-12-09T20:33:17.646 INFO:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2019-12-09T20:33:17.742 INFO:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2019-12-09T20:33:17.830 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-09T20:33:17.831 INFO:tasks.ceph:{} 2019-12-09T20:33:17.831 INFO:tasks.ceph:osd.2 2019-12-09T20:33:17.831 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:33:17.831 INFO:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-12-09T20:33:18.423 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-12-09T20:33:18.423 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-09T20:33:18.423 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=0, sparse=0 2019-12-09T20:33:18.423 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-09T20:33:18.424 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2019-12-09T20:33:18.424 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-09T20:33:18.424 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-09T20:33:18.424 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-09T20:33:18.424 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-09T20:33:18.427 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2019-12-09T20:33:18.427 INFO:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2019-12-09T20:33:18.610 INFO:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2019-12-09T20:33:18.726 INFO:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2019-12-09T20:33:18.823 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-09T20:33:18.823 INFO:tasks.ceph:{} 2019-12-09T20:33:18.823 INFO:tasks.ceph:osd.3 2019-12-09T20:33:18.823 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:33:18.823 INFO:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-12-09T20:33:19.387 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-12-09T20:33:19.387 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-09T20:33:19.388 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=0, sparse=0 2019-12-09T20:33:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-09T20:33:19.388 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2019-12-09T20:33:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-09T20:33:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-09T20:33:19.388 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-09T20:33:19.388 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-09T20:33:19.390 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2019-12-09T20:33:19.390 INFO:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2019-12-09T20:33:19.532 INFO:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2019-12-09T20:33:19.648 INFO:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:19.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:19.781+0000 7f844988fb40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2019-12-09T20:33:19.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:19.783+0000 7f844988fb40 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2019-12-09T20:33:19.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:19.783+0000 7f844988fb40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-09T20:33:19.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:19.784+0000 7f844988fb40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-09T20:33:19.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:19.784+0000 7f844988fb40 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-09T20:33:19.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:19.787+0000 7f844988fb40 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2019-12-09T20:33:22.113 INFO:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2019-12-09T20:33:22.240 INFO:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:22.392+0000 7f56c44ddb40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2019-12-09T20:33:22.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:22.393+0000 7f56c44ddb40 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2019-12-09T20:33:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:22.393+0000 7f56c44ddb40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-09T20:33:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:22.393+0000 7f56c44ddb40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-09T20:33:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:22.393+0000 7f56c44ddb40 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-09T20:33:22.409 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:22.394+0000 7f56c44ddb40 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2019-12-09T20:33:24.718 INFO:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2019-12-09T20:33:24.836 INFO:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:24.978 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:24.963+0000 7f866ea54b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2019-12-09T20:33:24.980 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:24.964+0000 7f866ea54b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2019-12-09T20:33:24.980 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:24.964+0000 7f866ea54b40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-09T20:33:24.980 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:24.964+0000 7f866ea54b40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-09T20:33:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:24.964+0000 7f866ea54b40 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-09T20:33:24.981 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:24.965+0000 7f866ea54b40 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2019-12-09T20:33:27.289 INFO:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2019-12-09T20:33:27.408 INFO:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:27.555 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:27.539+0000 7fabdeee3b40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-12-09T20:33:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:27.539+0000 7fabdeee3b40 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2019-12-09T20:33:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:27.539+0000 7fabdeee3b40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-09T20:33:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:27.539+0000 7fabdeee3b40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-09T20:33:27.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:27.539+0000 7fabdeee3b40 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-09T20:33:27.557 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:33:27.541+0000 7fabdeee3b40 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2019-12-09T20:33:29.864 INFO:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2019-12-09T20:33:29.979 INFO:tasks.ceph:Reading keys from all nodes... 2019-12-09T20:33:29.980 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:33:30.075 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.wO0kPyQt2x 2019-12-09T20:33:30.075 INFO:teuthology.orchestra.run.smithi081:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.wO0kPyQt2x 2019-12-09T20:33:30.182 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.wO0kPyQt2x 2019-12-09T20:33:30.327 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.wO0kPyQt2x is 56B 2019-12-09T20:33:30.390 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.wO0kPyQt2x 2019-12-09T20:33:30.444 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:33:30.558 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.tRaTE1goPG 2019-12-09T20:33:30.562 INFO:teuthology.orchestra.run.smithi081:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.tRaTE1goPG 2019-12-09T20:33:30.665 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.tRaTE1goPG 2019-12-09T20:33:30.841 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.tRaTE1goPG is 56B 2019-12-09T20:33:30.907 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.tRaTE1goPG 2019-12-09T20:33:30.969 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:33:31.020 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.6UVqMjl7zO 2019-12-09T20:33:31.021 INFO:teuthology.orchestra.run.smithi081:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.6UVqMjl7zO 2019-12-09T20:33:31.093 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.6UVqMjl7zO 2019-12-09T20:33:31.215 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.6UVqMjl7zO is 56B 2019-12-09T20:33:31.269 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.6UVqMjl7zO 2019-12-09T20:33:31.316 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:33:31.363 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.4hzAnh1dRc 2019-12-09T20:33:31.364 INFO:teuthology.orchestra.run.smithi081:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.4hzAnh1dRc 2019-12-09T20:33:31.429 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.4hzAnh1dRc 2019-12-09T20:33:31.539 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.4hzAnh1dRc is 56B 2019-12-09T20:33:31.594 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.4hzAnh1dRc 2019-12-09T20:33:31.638 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:33:31.747 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.N8SSRjYJJI 2019-12-09T20:33:31.748 INFO:teuthology.orchestra.run.smithi081:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.N8SSRjYJJI 2019-12-09T20:33:31.834 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.N8SSRjYJJI 2019-12-09T20:33:31.962 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.N8SSRjYJJI is 56B 2019-12-09T20:33:32.017 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.N8SSRjYJJI 2019-12-09T20:33:32.065 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:33:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.t7lph12P67 2019-12-09T20:33:32.157 INFO:teuthology.orchestra.run.smithi067:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.t7lph12P67 2019-12-09T20:33:32.274 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.t7lph12P67 2019-12-09T20:33:32.467 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.t7lph12P67 is 56B 2019-12-09T20:33:32.563 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.t7lph12P67 2019-12-09T20:33:32.649 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:33:32.736 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.1keapmoeW2 2019-12-09T20:33:32.736 INFO:teuthology.orchestra.run.smithi067:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.1keapmoeW2 2019-12-09T20:33:32.845 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.1keapmoeW2 2019-12-09T20:33:33.047 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.1keapmoeW2 is 56B 2019-12-09T20:33:33.133 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.1keapmoeW2 2019-12-09T20:33:33.215 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:33:33.288 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.O0X57F7cdA 2019-12-09T20:33:33.289 INFO:teuthology.orchestra.run.smithi067:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.O0X57F7cdA 2019-12-09T20:33:33.392 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.O0X57F7cdA 2019-12-09T20:33:33.563 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.O0X57F7cdA is 56B 2019-12-09T20:33:33.672 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.O0X57F7cdA 2019-12-09T20:33:33.758 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:33:33.890 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.0lI92qk0rr 2019-12-09T20:33:33.890 INFO:teuthology.orchestra.run.smithi067:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.0lI92qk0rr 2019-12-09T20:33:33.999 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.0lI92qk0rr 2019-12-09T20:33:34.207 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.0lI92qk0rr is 56B 2019-12-09T20:33:34.286 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.0lI92qk0rr 2019-12-09T20:33:34.356 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:33:34.422 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.U0Woh21au2 2019-12-09T20:33:34.423 INFO:teuthology.orchestra.run.smithi067:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.U0Woh21au2 2019-12-09T20:33:34.525 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.U0Woh21au2 2019-12-09T20:33:34.731 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.U0Woh21au2 is 56B 2019-12-09T20:33:34.809 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.U0Woh21au2 2019-12-09T20:33:34.973 DEBUG:teuthology.orchestra.remote:smithi081:/etc/ceph/ceph.client.1.keyring is 59B 2019-12-09T20:33:35.153 DEBUG:teuthology.orchestra.remote:smithi067:/etc/ceph/ceph.client.0.keyring is 59B 2019-12-09T20:33:35.248 INFO:tasks.ceph:Adding keys to all mons... 2019-12-09T20:33:35.248 INFO:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2019-12-09T20:33:35.253 INFO:teuthology.orchestra.run.smithi081:> sudo tee -a /etc/ceph/ceph.keyring 2019-12-09T20:33:35.353 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2019-12-09T20:33:35.353 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB9r+5dB21DBhAAgmLkSDuT7I2NgiXjoMtBqw== 2019-12-09T20:33:35.353 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2019-12-09T20:33:35.353 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCBr+5du0VfJBAAHoGCMMoNjJ0HbW3tUdJU6A== 2019-12-09T20:33:35.353 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2019-12-09T20:33:35.354 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCEr+5daBsEChAAruQMGbuTk7po64P48uTs/w== 2019-12-09T20:33:35.354 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2019-12-09T20:33:35.354 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCGr+5dkArWKxAAPVMpNv7zfRWDePZCWrCTbA== 2019-12-09T20:33:35.354 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2019-12-09T20:33:35.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCJr+5dNbTcEhAAiWUfw39Iks5VjyQ16TGoRw== 2019-12-09T20:33:35.355 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2019-12-09T20:33:35.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB9r+5dhMWkEBAAst5pspsZkaCF8fRM44SXNg== 2019-12-09T20:33:35.355 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2019-12-09T20:33:35.355 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCPr+5dT5edLhAABwipBUvFRZjJnjTolewl3A== 2019-12-09T20:33:35.356 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2019-12-09T20:33:35.356 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCSr+5deqxvFxAAHzaIIxXN8a8XFy1m1dyjBg== 2019-12-09T20:33:35.356 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2019-12-09T20:33:35.356 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCUr+5dK+Z4ORAAdee45WS4Pvk/S48c/UVOEA== 2019-12-09T20:33:35.356 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2019-12-09T20:33:35.356 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCXr+5duNk8IBAAU0AreHpl2G5X+OBOSwadGg== 2019-12-09T20:33:35.356 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2019-12-09T20:33:35.357 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB9r+5dtbXDGhAAn7DLWxWZnqgzSWF2uFwxMA== 2019-12-09T20:33:35.357 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2019-12-09T20:33:35.357 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQB9r+5d5j9mJRAAtUFJP6NgrzvFZ9xHRM0gAQ== 2019-12-09T20:33:35.366 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.x] 2019-12-09T20:33:35.366 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB9r+5dB21DBhAAgmLkSDuT7I2NgiXjoMtBqw== 2019-12-09T20:33:35.367 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.4] 2019-12-09T20:33:35.367 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCBr+5du0VfJBAAHoGCMMoNjJ0HbW3tUdJU6A== 2019-12-09T20:33:35.367 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.5] 2019-12-09T20:33:35.367 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCEr+5daBsEChAAruQMGbuTk7po64P48uTs/w== 2019-12-09T20:33:35.367 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.6] 2019-12-09T20:33:35.367 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCGr+5dkArWKxAAPVMpNv7zfRWDePZCWrCTbA== 2019-12-09T20:33:35.367 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.7] 2019-12-09T20:33:35.368 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCJr+5dNbTcEhAAiWUfw39Iks5VjyQ16TGoRw== 2019-12-09T20:33:35.368 INFO:teuthology.orchestra.run.smithi081.stdout:[mgr.y] 2019-12-09T20:33:35.368 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB9r+5dhMWkEBAAst5pspsZkaCF8fRM44SXNg== 2019-12-09T20:33:35.368 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.0] 2019-12-09T20:33:35.368 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCPr+5dT5edLhAABwipBUvFRZjJnjTolewl3A== 2019-12-09T20:33:35.368 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.1] 2019-12-09T20:33:35.368 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCSr+5deqxvFxAAHzaIIxXN8a8XFy1m1dyjBg== 2019-12-09T20:33:35.369 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.2] 2019-12-09T20:33:35.369 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCUr+5dK+Z4ORAAdee45WS4Pvk/S48c/UVOEA== 2019-12-09T20:33:35.369 INFO:teuthology.orchestra.run.smithi081.stdout:[osd.3] 2019-12-09T20:33:35.369 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQCXr+5duNk8IBAAU0AreHpl2G5X+OBOSwadGg== 2019-12-09T20:33:35.369 INFO:teuthology.orchestra.run.smithi081.stdout:[client.1] 2019-12-09T20:33:35.369 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB9r+5dtbXDGhAAn7DLWxWZnqgzSWF2uFwxMA== 2019-12-09T20:33:35.369 INFO:teuthology.orchestra.run.smithi081.stdout:[client.0] 2019-12-09T20:33:35.370 INFO:teuthology.orchestra.run.smithi081.stdout: key = AQB9r+5d5j9mJRAAtUFJP6NgrzvFZ9xHRM0gAQ== 2019-12-09T20:33:35.371 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-09T20:33:35.374 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-09T20:33:35.514 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:35.517 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:35.684 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:35.688 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:35.843 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:35.847 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.014 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.019 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.185 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-09T20:33:36.190 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-09T20:33:36.357 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.364 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.527 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.531 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.694 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.700 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.870 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:36.874 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-09T20:33:37.047 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-09T20:33:37.051 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-09T20:33:37.234 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-09T20:33:37.238 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-09T20:33:37.409 INFO:tasks.ceph:Running mkfs on mon nodes... 2019-12-09T20:33:37.409 INFO:teuthology.orchestra.run.smithi081:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2019-12-09T20:33:37.526 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-09T20:33:37.663 INFO:teuthology.orchestra.run.smithi081:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2019-12-09T20:33:37.736 INFO:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2019-12-09T20:33:37.855 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-09T20:33:38.034 INFO:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2019-12-09T20:33:38.139 INFO:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2019-12-09T20:33:38.249 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-09T20:33:38.427 INFO:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2019-12-09T20:33:38.548 INFO:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:38.552 INFO:teuthology.orchestra.run.smithi081:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-12-09T20:33:38.643 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2019-12-09T20:33:38.643 DEBUG:teuthology.misc:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2019-12-09T20:33:38.643 INFO:tasks.ceph.mon.b:Restarting daemon 2019-12-09T20:33:38.643 INFO:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2019-12-09T20:33:38.647 INFO:tasks.ceph.mon.b:Started 2019-12-09T20:33:38.647 DEBUG:teuthology.misc:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2019-12-09T20:33:38.647 INFO:tasks.ceph.mon.a:Restarting daemon 2019-12-09T20:33:38.648 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2019-12-09T20:33:38.651 INFO:tasks.ceph.mon.a:Started 2019-12-09T20:33:38.652 DEBUG:teuthology.misc:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2019-12-09T20:33:38.652 INFO:tasks.ceph.mon.c:Restarting daemon 2019-12-09T20:33:38.652 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2019-12-09T20:33:38.655 INFO:tasks.ceph.mon.c:Started 2019-12-09T20:33:38.655 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2019-12-09T20:33:38.656 INFO:tasks.ceph.mgr.x:Restarting daemon 2019-12-09T20:33:38.656 INFO:teuthology.orchestra.run.smithi081:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2019-12-09T20:33:38.659 INFO:tasks.ceph.mgr.x:Started 2019-12-09T20:33:38.659 INFO:tasks.ceph.mgr.y:Restarting daemon 2019-12-09T20:33:38.660 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2019-12-09T20:33:38.664 INFO:tasks.ceph.mgr.y:Started 2019-12-09T20:33:38.664 INFO:tasks.ceph:Setting crush tunables to default 2019-12-09T20:33:38.664 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2019-12-09T20:33:42.732 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:00:03.896 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:33:42.860 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:00:03.991 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:33:42.886 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:00:04.017 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:33:55.057 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2019-12-09T20:33:55.072 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2019-12-09T20:33:55.073 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:33:55.158 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.iWNlrHGO1r 2019-12-09T20:33:55.158 INFO:teuthology.orchestra.run.smithi081:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.iWNlrHGO1r 2019-12-09T20:33:55.238 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.iWNlrHGO1r 2019-12-09T20:33:55.361 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.iWNlrHGO1r is 37B 2019-12-09T20:33:55.409 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.iWNlrHGO1r 2019-12-09T20:33:55.457 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:33:55.548 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.WHUk8nigTZ 2019-12-09T20:33:55.549 INFO:teuthology.orchestra.run.smithi081:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.WHUk8nigTZ 2019-12-09T20:33:55.606 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.WHUk8nigTZ 2019-12-09T20:33:55.709 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.WHUk8nigTZ is 37B 2019-12-09T20:33:55.757 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.WHUk8nigTZ 2019-12-09T20:33:55.799 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:33:55.886 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.YFxJqBNt5C 2019-12-09T20:33:55.886 INFO:teuthology.orchestra.run.smithi081:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.YFxJqBNt5C 2019-12-09T20:33:55.942 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.YFxJqBNt5C 2019-12-09T20:33:56.043 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.YFxJqBNt5C is 37B 2019-12-09T20:33:56.094 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.YFxJqBNt5C 2019-12-09T20:33:56.143 INFO:teuthology.orchestra.run.smithi081:> mktemp 2019-12-09T20:33:56.248 INFO:teuthology.orchestra.run.smithi081.stdout:/tmp/tmp.jyvn9KsAQ0 2019-12-09T20:33:56.249 INFO:teuthology.orchestra.run.smithi081:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.jyvn9KsAQ0 2019-12-09T20:33:56.331 INFO:teuthology.orchestra.run.smithi081:> sudo chmod 0666 /tmp/tmp.jyvn9KsAQ0 2019-12-09T20:33:56.458 DEBUG:teuthology.orchestra.remote:smithi081:/tmp/tmp.jyvn9KsAQ0 is 37B 2019-12-09T20:33:56.511 INFO:teuthology.orchestra.run.smithi081:> rm -fr /tmp/tmp.jyvn9KsAQ0 2019-12-09T20:33:56.554 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:33:56.644 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.Bnfw9rPIk3 2019-12-09T20:33:56.645 INFO:teuthology.orchestra.run.smithi067:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.Bnfw9rPIk3 2019-12-09T20:33:56.741 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.Bnfw9rPIk3 2019-12-09T20:33:56.866 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.Bnfw9rPIk3 is 37B 2019-12-09T20:33:56.915 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.Bnfw9rPIk3 2019-12-09T20:33:56.958 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:33:57.047 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.MOUfnqhm7X 2019-12-09T20:33:57.047 INFO:teuthology.orchestra.run.smithi067:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.MOUfnqhm7X 2019-12-09T20:33:57.104 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.MOUfnqhm7X 2019-12-09T20:33:57.214 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.MOUfnqhm7X is 37B 2019-12-09T20:33:57.262 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.MOUfnqhm7X 2019-12-09T20:33:57.305 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:33:57.398 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.7lDfzWyi9t 2019-12-09T20:33:57.398 INFO:teuthology.orchestra.run.smithi067:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.7lDfzWyi9t 2019-12-09T20:33:57.461 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.7lDfzWyi9t 2019-12-09T20:33:57.561 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.7lDfzWyi9t is 37B 2019-12-09T20:33:57.611 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.7lDfzWyi9t 2019-12-09T20:33:57.653 INFO:teuthology.orchestra.run.smithi067:> mktemp 2019-12-09T20:33:57.759 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.mEv2NEWapo 2019-12-09T20:33:57.759 INFO:teuthology.orchestra.run.smithi067:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.mEv2NEWapo 2019-12-09T20:33:57.846 INFO:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.mEv2NEWapo 2019-12-09T20:33:57.966 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.mEv2NEWapo is 37B 2019-12-09T20:33:58.014 INFO:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.mEv2NEWapo 2019-12-09T20:33:58.061 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new bcb07227-f2ab-47ba-97ac-3302ac851466 0 2019-12-09T20:33:59.269 INFO:teuthology.orchestra.run.smithi067.stdout:0 2019-12-09T20:33:59.296 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 194d5481-21e3-4653-9ca6-8154ace63907 1 2019-12-09T20:34:00.342 INFO:teuthology.orchestra.run.smithi067.stdout:1 2019-12-09T20:34:00.357 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new ddcb83a9-ab24-4e56-a27c-d75f3b589c69 2 2019-12-09T20:34:01.371 INFO:teuthology.orchestra.run.smithi067.stdout:2 2019-12-09T20:34:01.392 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new bde673fa-eeb4-4878-bbde-0ec9ae07d78c 3 2019-12-09T20:34:02.370 INFO:teuthology.orchestra.run.smithi067.stdout:3 2019-12-09T20:34:02.395 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new c5f858b6-a265-49c9-8120-c594db9230c2 4 2019-12-09T20:34:03.403 INFO:teuthology.orchestra.run.smithi067.stdout:4 2019-12-09T20:34:03.423 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new e2d52d92-a27b-4582-8713-171ce412b4a7 5 2019-12-09T20:34:04.417 INFO:teuthology.orchestra.run.smithi067.stdout:5 2019-12-09T20:34:04.438 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new 2f23a442-4fc3-417f-aeca-c0d9130e5f85 6 2019-12-09T20:34:05.405 INFO:teuthology.orchestra.run.smithi067.stdout:6 2019-12-09T20:34:05.425 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd new bb0ada9f-667e-403a-8317-e2ddc59c7dee 7 2019-12-09T20:34:06.267 INFO:teuthology.orchestra.run.smithi067.stdout:7 2019-12-09T20:34:06.286 DEBUG:teuthology.misc:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-09T20:34:06.287 INFO:tasks.ceph.osd.4:Restarting daemon 2019-12-09T20:34:06.287 INFO:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 4 2019-12-09T20:34:06.293 INFO:tasks.ceph.osd.4:Started 2019-12-09T20:34:06.294 DEBUG:teuthology.misc:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-09T20:34:06.294 INFO:tasks.ceph.osd.5:Restarting daemon 2019-12-09T20:34:06.294 INFO:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 5 2019-12-09T20:34:06.298 INFO:tasks.ceph.osd.5:Started 2019-12-09T20:34:06.298 DEBUG:teuthology.misc:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-09T20:34:06.299 INFO:tasks.ceph.osd.6:Restarting daemon 2019-12-09T20:34:06.299 INFO:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 6 2019-12-09T20:34:06.302 INFO:tasks.ceph.osd.6:Started 2019-12-09T20:34:06.302 DEBUG:teuthology.misc:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-09T20:34:06.302 INFO:tasks.ceph.osd.7:Restarting daemon 2019-12-09T20:34:06.302 INFO:teuthology.orchestra.run.smithi081:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 7 2019-12-09T20:34:06.305 INFO:tasks.ceph.osd.7:Started 2019-12-09T20:34:06.305 DEBUG:teuthology.misc:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-09T20:34:06.305 INFO:tasks.ceph.osd.0:Restarting daemon 2019-12-09T20:34:06.305 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 0 2019-12-09T20:34:06.309 INFO:tasks.ceph.osd.0:Started 2019-12-09T20:34:06.309 DEBUG:teuthology.misc:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-09T20:34:06.309 INFO:tasks.ceph.osd.1:Restarting daemon 2019-12-09T20:34:06.309 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 1 2019-12-09T20:34:06.312 INFO:tasks.ceph.osd.1:Started 2019-12-09T20:34:06.312 DEBUG:teuthology.misc:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-09T20:34:06.312 INFO:tasks.ceph.osd.2:Restarting daemon 2019-12-09T20:34:06.313 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 2 2019-12-09T20:34:06.316 INFO:tasks.ceph.osd.2:Started 2019-12-09T20:34:06.316 DEBUG:teuthology.misc:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--tool=memcheck'] 2019-12-09T20:34:06.316 INFO:tasks.ceph.osd.3:Restarting daemon 2019-12-09T20:34:06.316 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 3 2019-12-09T20:34:06.319 INFO:tasks.ceph.osd.3:Started 2019-12-09T20:34:06.319 INFO:tasks.ceph:Waiting for OSDs to come up 2019-12-09T20:34:07.130 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-09T20:34:08.084 INFO:teuthology.misc.health.smithi067.stdout: 2019-12-09T20:34:08.085 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"bb0ccc8e-29f2-48f9-838e-c4ccdeefe22d","created":"2019-12-09T20:33:52.288217+0000","modified":"2019-12-09T20:34:06.099585+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"bcb07227-f2ab-47ba-97ac-3302ac851466","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"194d5481-21e3-4653-9ca6-8154ace63907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddcb83a9-ab24-4e56-a27c-d75f3b589c69","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bde673fa-eeb4-4878-bbde-0ec9ae07d78c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c5f858b6-a265-49c9-8120-c594db9230c2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2d52d92-a27b-4582-8713-171ce412b4a7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f23a442-4fc3-417f-aeca-c0d9130e5f85","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bb0ada9f-667e-403a-8317-e2ddc59c7dee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-09T20:34:08.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-12-09T20:34:10.059 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:03.623 9680== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.126 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:03.689 9680== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.161 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:03.737 9678== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.174 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:03.726 9681== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.191 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:03.764 9679== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.219 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:03.795 9678== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.256 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:03.808 9681== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.272 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:03.844 9679== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.299 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:03.839 10705== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.344 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:03.891 10703== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.357 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:03.896 10705== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.402 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:03.949 10703== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.734 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-09T20:34:10.709+0000 9b13800 -1 Falling back to public interface 2019-12-09T20:34:10.849 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-09T20:34:10.835+0000 9b13800 -1 Falling back to public interface 2019-12-09T20:34:10.867 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-09T20:34:10.851+0000 9b13800 -1 Falling back to public interface 2019-12-09T20:34:10.875 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-09T20:34:10.864+0000 9b13800 -1 Falling back to public interface 2019-12-09T20:34:10.937 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:04.501 9680== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:10.941 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-09T20:34:10.908+0000 9b13800 -1 Falling back to public interface 2019-12-09T20:34:10.950 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-09T20:34:10.920+0000 9b13800 -1 Falling back to public interface 2019-12-09T20:34:11.066 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:04.643 9678== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.079 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:04.651 9679== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.092 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:04.644 9681== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.136 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:04.681 10703== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.159 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:04.703 10704== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.213 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:04.785 9679== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.215 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:04.762 10704== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.216 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:04.746 10706== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.224 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:04.797 9679== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.247 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:04.786 10705== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.258 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:04.805 10703== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.275 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:04.822 10703== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.289 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:04.819 10706== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.369 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:04.928 9680== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.386 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:04.950 9680== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.460 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:05.011 9681== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.473 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:05.025 9681== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.488 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:05.065 9678== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.505 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:05.082 9678== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.658 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:05.197 10705== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.670 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:05.210 10705== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:11.837 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-09T20:34:11.803+0000 9b13800 -1 Falling back to public interface 2019-12-09T20:34:11.895 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-09T20:34:11.863+0000 9b13800 -1 Falling back to public interface 2019-12-09T20:34:12.110 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:05.639 10706== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:12.156 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:05.703 10704== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:12.514 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:06.044 10706== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:12.529 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:06.059 10706== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:12.576 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:06.122 10704== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:12.594 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:06.140 10704== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.354 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:07.893 10705== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.358 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:07.898 10705== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.424 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:07.976 9681== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.431 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:07.983 9681== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.485 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:08.049 9680== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.491 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:08.054 9680== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.526 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:08.073 10703== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.535 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:08.083 10703== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.623 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:08.162 10705== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.693 INFO:tasks.ceph.osd.4.smithi081.stderr:==00:00:00:08.245 9681== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.750 INFO:tasks.ceph.osd.6.smithi081.stderr:==00:00:00:08.314 9680== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.800 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:08.347 10703== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.891 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:08.467 9678== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.901 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:08.478 9678== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.904 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-09T20:34:14.982 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:08.552 9679== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:14.991 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:08.563 9679== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:15.161 INFO:tasks.ceph.osd.7.smithi081.stderr:==00:00:00:08.738 9678== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:15.258 INFO:tasks.ceph.osd.5.smithi081.stderr:==00:00:00:08.830 9679== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:15.600 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:09.147 10704== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:15.611 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:09.157 10704== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:15.656 INFO:teuthology.misc.health.smithi067.stdout: 2019-12-09T20:34:15.774 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"bb0ccc8e-29f2-48f9-838e-c4ccdeefe22d","created":"2019-12-09T20:33:52.288217+0000","modified":"2019-12-09T20:34:06.099585+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"bcb07227-f2ab-47ba-97ac-3302ac851466","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"194d5481-21e3-4653-9ca6-8154ace63907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddcb83a9-ab24-4e56-a27c-d75f3b589c69","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"bde673fa-eeb4-4878-bbde-0ec9ae07d78c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c5f858b6-a265-49c9-8120-c594db9230c2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e2d52d92-a27b-4582-8713-171ce412b4a7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2f23a442-4fc3-417f-aeca-c0d9130e5f85","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"bb0ada9f-667e-403a-8317-e2ddc59c7dee","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-09T20:34:15.776 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:09.193 10706== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:15.776 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:09.201 10706== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:15.777 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-12-09T20:34:15.874 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:09.419 10704== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:15.937 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:09.466 10706== Warning: unimplemented fcntl command: 1038 2019-12-09T20:34:16.105 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-09T20:34:16.084+0000 9b13800 -1 osd.2 0 log_to_monitors {default=true} 2019-12-09T20:34:16.117 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-09T20:34:16.115+0000 9b13800 -1 osd.4 0 log_to_monitors {default=true} 2019-12-09T20:34:16.168 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-09T20:34:16.167+0000 9b13800 -1 osd.6 0 log_to_monitors {default=true} 2019-12-09T20:34:16.422 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-09T20:34:16.402+0000 9b13800 -1 osd.0 0 log_to_monitors {default=true} 2019-12-09T20:34:16.531 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-09T20:34:16.531+0000 9b13800 -1 osd.7 0 log_to_monitors {default=true} 2019-12-09T20:34:16.641 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-09T20:34:16.641+0000 9b13800 -1 osd.5 0 log_to_monitors {default=true} 2019-12-09T20:34:17.291 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-09T20:34:17.269+0000 9b13800 -1 osd.3 0 log_to_monitors {default=true} 2019-12-09T20:34:17.347 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-09T20:34:17.325+0000 9b13800 -1 osd.1 0 log_to_monitors {default=true} 2019-12-09T20:34:18.513 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-09T20:34:18.489+0000 27772700 -1 osd.0 0 waiting for initial osdmap 2019-12-09T20:34:18.524 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-09T20:34:18.521+0000 2476e700 -1 osd.7 0 waiting for initial osdmap 2019-12-09T20:34:18.531 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-09T20:34:18.508+0000 2476e700 -1 osd.2 0 waiting for initial osdmap 2019-12-09T20:34:18.572 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-09T20:34:18.569+0000 27772700 -1 osd.6 0 waiting for initial osdmap 2019-12-09T20:34:18.594 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-09T20:34:18.591+0000 27772700 -1 osd.4 0 waiting for initial osdmap 2019-12-09T20:34:19.766 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-09T20:34:19.759+0000 2476e700 -1 osd.5 0 waiting for initial osdmap 2019-12-09T20:34:19.874 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-09T20:34:19.849+0000 27772700 -1 osd.1 0 waiting for initial osdmap 2019-12-09T20:34:19.891 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-09T20:34:19.870+0000 2476e700 -1 osd.3 0 waiting for initial osdmap 2019-12-09T20:34:22.580 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-09T20:34:23.496 INFO:teuthology.misc.health.smithi067.stdout: 2019-12-09T20:34:23.496 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":16,"fsid":"bb0ccc8e-29f2-48f9-838e-c4ccdeefe22d","created":"2019-12-09T20:33:52.288217+0000","modified":"2019-12-09T20:34:22.574459+0000","last_up_change":"2019-12-09T20:34:21.509278+0000","last_in_change":"2019-12-09T20:34:21.509278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"bcb07227-f2ab-47ba-97ac-3302ac851466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":10703},{"type":"v1","addr":"172.21.15.67:6806","nonce":10703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":10703},{"type":"v1","addr":"172.21.15.67:6810","nonce":10703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":10703},{"type":"v1","addr":"172.21.15.67:6817","nonce":10703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":10703},{"type":"v1","addr":"172.21.15.67:6813","nonce":10703}]},"public_addr":"172.21.15.67:6806/10703","cluster_addr":"172.21.15.67:6810/10703","heartbeat_back_addr":"172.21.15.67:6817/10703","heartbeat_front_addr":"172.21.15.67:6813/10703","state":["exists","up"]},{"osd":1,"uuid":"194d5481-21e3-4653-9ca6-8154ace63907","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":10706},{"type":"v1","addr":"172.21.15.67:6827","nonce":10706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":10706},{"type":"v1","addr":"172.21.15.67:6829","nonce":10706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":10706},{"type":"v1","addr":"172.21.15.67:6833","nonce":10706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":10706},{"type":"v1","addr":"172.21.15.67:6831","nonce":10706}]},"public_addr":"172.21.15.67:6827/10706","cluster_addr":"172.21.15.67:6829/10706","heartbeat_back_addr":"172.21.15.67:6833/10706","heartbeat_front_addr":"172.21.15.67:6831/10706","state":["exists","up"]},{"osd":2,"uuid":"ddcb83a9-ab24-4e56-a27c-d75f3b589c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":10705},{"type":"v1","addr":"172.21.15.67:6803","nonce":10705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":10705},{"type":"v1","addr":"172.21.15.67:6807","nonce":10705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":10705},{"type":"v1","addr":"172.21.15.67:6816","nonce":10705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":10705},{"type":"v1","addr":"172.21.15.67:6811","nonce":10705}]},"public_addr":"172.21.15.67:6803/10705","cluster_addr":"172.21.15.67:6807/10705","heartbeat_back_addr":"172.21.15.67:6816/10705","heartbeat_front_addr":"172.21.15.67:6811/10705","state":["exists","up"]},{"osd":3,"uuid":"bde673fa-eeb4-4878-bbde-0ec9ae07d78c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":10704},{"type":"v1","addr":"172.21.15.67:6819","nonce":10704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":10704},{"type":"v1","addr":"172.21.15.67:6821","nonce":10704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":10704},{"type":"v1","addr":"172.21.15.67:6825","nonce":10704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":10704},{"type":"v1","addr":"172.21.15.67:6823","nonce":10704}]},"public_addr":"172.21.15.67:6819/10704","cluster_addr":"172.21.15.67:6821/10704","heartbeat_back_addr":"172.21.15.67:6825/10704","heartbeat_front_addr":"172.21.15.67:6823/10704","state":["exists","up"]},{"osd":4,"uuid":"c5f858b6-a265-49c9-8120-c594db9230c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":9681},{"type":"v1","addr":"172.21.15.81:6818","nonce":9681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":9681},{"type":"v1","addr":"172.21.15.81:6825","nonce":9681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":9681},{"type":"v1","addr":"172.21.15.81:6831","nonce":9681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":9681},{"type":"v1","addr":"172.21.15.81:6829","nonce":9681}]},"public_addr":"172.21.15.81:6818/9681","cluster_addr":"172.21.15.81:6825/9681","heartbeat_back_addr":"172.21.15.81:6831/9681","heartbeat_front_addr":"172.21.15.81:6829/9681","state":["exists","up"]},{"osd":5,"uuid":"e2d52d92-a27b-4582-8713-171ce412b4a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":9679},{"type":"v1","addr":"172.21.15.81:6814","nonce":9679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":9679},{"type":"v1","addr":"172.21.15.81:6821","nonce":9679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":9679},{"type":"v1","addr":"172.21.15.81:6827","nonce":9679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":9679},{"type":"v1","addr":"172.21.15.81:6824","nonce":9679}]},"public_addr":"172.21.15.81:6814/9679","cluster_addr":"172.21.15.81:6821/9679","heartbeat_back_addr":"172.21.15.81:6827/9679","heartbeat_front_addr":"172.21.15.81:6824/9679","state":["exists","up"]},{"osd":6,"uuid":"2f23a442-4fc3-417f-aeca-c0d9130e5f85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":9680},{"type":"v1","addr":"172.21.15.81:6801","nonce":9680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":9680},{"type":"v1","addr":"172.21.15.81:6803","nonce":9680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":9680},{"type":"v1","addr":"172.21.15.81:6807","nonce":9680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":9680},{"type":"v1","addr":"172.21.15.81:6805","nonce":9680}]},"public_addr":"172.21.15.81:6801/9680","cluster_addr":"172.21.15.81:6803/9680","heartbeat_back_addr":"172.21.15.81:6807/9680","heartbeat_front_addr":"172.21.15.81:6805/9680","state":["exists","up"]},{"osd":7,"uuid":"bb0ada9f-667e-403a-8317-e2ddc59c7dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":9678},{"type":"v1","addr":"172.21.15.81:6809","nonce":9678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":9678},{"type":"v1","addr":"172.21.15.81:6811","nonce":9678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":9678},{"type":"v1","addr":"172.21.15.81:6820","nonce":9678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":9678},{"type":"v1","addr":"172.21.15.81:6815","nonce":9678}]},"public_addr":"172.21.15.81:6809/9678","cluster_addr":"172.21.15.81:6811/9678","heartbeat_back_addr":"172.21.15.81:6820/9678","heartbeat_front_addr":"172.21.15.81:6815/9678","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.455163+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:18.395441+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.217011+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:18.335336+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.237049+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.289402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.600264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-09T20:34:23.520 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-12-09T20:34:23.520 INFO:tasks.ceph:Creating RBD pool 2019-12-09T20:34:23.520 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd pool create rbd 8 2019-12-09T20:34:26.141 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'rbd' created 2019-12-09T20:34:26.155 INFO:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2019-12-09T20:34:28.045 INFO:teuthology.orchestra.run.smithi067.stderr:enabled application 'rbd' on pool 'rbd' 2019-12-09T20:34:28.062 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2019-12-09T20:34:28.063 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-09T20:34:28.064 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2019-12-09T20:34:28.993 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:34:28.993 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":20,"fsid":"bb0ccc8e-29f2-48f9-838e-c4ccdeefe22d","created":"2019-12-09T20:33:52.288217+0000","modified":"2019-12-09T20:34:27.890325+0000","last_up_change":"2019-12-09T20:34:21.509278+0000","last_in_change":"2019-12-09T20:34:21.509278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-09T20:34:24.453766+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bcb07227-f2ab-47ba-97ac-3302ac851466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":10703},{"type":"v1","addr":"172.21.15.67:6806","nonce":10703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":10703},{"type":"v1","addr":"172.21.15.67:6810","nonce":10703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":10703},{"type":"v1","addr":"172.21.15.67:6817","nonce":10703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":10703},{"type":"v1","addr":"172.21.15.67:6813","nonce":10703}]},"public_addr":"172.21.15.67:6806/10703","cluster_addr":"172.21.15.67:6810/10703","heartbeat_back_addr":"172.21.15.67:6817/10703","heartbeat_front_addr":"172.21.15.67:6813/10703","state":["exists","up"]},{"osd":1,"uuid":"194d5481-21e3-4653-9ca6-8154ace63907","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":10706},{"type":"v1","addr":"172.21.15.67:6827","nonce":10706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":10706},{"type":"v1","addr":"172.21.15.67:6829","nonce":10706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":10706},{"type":"v1","addr":"172.21.15.67:6833","nonce":10706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":10706},{"type":"v1","addr":"172.21.15.67:6831","nonce":10706}]},"public_addr":"172.21.15.67:6827/10706","cluster_addr":"172.21.15.67:6829/10706","heartbeat_back_addr":"172.21.15.67:6833/10706","heartbeat_front_addr":"172.21.15.67:6831/10706","state":["exists","up"]},{"osd":2,"uuid":"ddcb83a9-ab24-4e56-a27c-d75f3b589c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":10705},{"type":"v1","addr":"172.21.15.67:6803","nonce":10705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":10705},{"type":"v1","addr":"172.21.15.67:6807","nonce":10705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":10705},{"type":"v1","addr":"172.21.15.67:6816","nonce":10705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":10705},{"type":"v1","addr":"172.21.15.67:6811","nonce":10705}]},"public_addr":"172.21.15.67:6803/10705","cluster_addr":"172.21.15.67:6807/10705","heartbeat_back_addr":"172.21.15.67:6816/10705","heartbeat_front_addr":"172.21.15.67:6811/10705","state":["exists","up"]},{"osd":3,"uuid":"bde673fa-eeb4-4878-bbde-0ec9ae07d78c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":10704},{"type":"v1","addr":"172.21.15.67:6819","nonce":10704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":10704},{"type":"v1","addr":"172.21.15.67:6821","nonce":10704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":10704},{"type":"v1","addr":"172.21.15.67:6825","nonce":10704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":10704},{"type":"v1","addr":"172.21.15.67:6823","nonce":10704}]},"public_addr":"172.21.15.67:6819/10704","cluster_addr":"172.21.15.67:6821/10704","heartbeat_back_addr":"172.21.15.67:6825/10704","heartbeat_front_addr":"172.21.15.67:6823/10704","state":["exists","up"]},{"osd":4,"uuid":"c5f858b6-a265-49c9-8120-c594db9230c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":9681},{"type":"v1","addr":"172.21.15.81:6818","nonce":9681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":9681},{"type":"v1","addr":"172.21.15.81:6825","nonce":9681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":9681},{"type":"v1","addr":"172.21.15.81:6831","nonce":9681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":9681},{"type":"v1","addr":"172.21.15.81:6829","nonce":9681}]},"public_addr":"172.21.15.81:6818/9681","cluster_addr":"172.21.15.81:6825/9681","heartbeat_back_addr":"172.21.15.81:6831/9681","heartbeat_front_addr":"172.21.15.81:6829/9681","state":["exists","up"]},{"osd":5,"uuid":"e2d52d92-a27b-4582-8713-171ce412b4a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":9679},{"type":"v1","addr":"172.21.15.81:6814","nonce":9679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":9679},{"type":"v1","addr":"172.21.15.81:6821","nonce":9679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":9679},{"type":"v1","addr":"172.21.15.81:6827","nonce":9679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":9679},{"type":"v1","addr":"172.21.15.81:6824","nonce":9679}]},"public_addr":"172.21.15.81:6814/9679","cluster_addr":"172.21.15.81:6821/9679","heartbeat_back_addr":"172.21.15.81:6827/9679","heartbeat_front_addr":"172.21.15.81:6824/9679","state":["exists","up"]},{"osd":6,"uuid":"2f23a442-4fc3-417f-aeca-c0d9130e5f85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":9680},{"type":"v1","addr":"172.21.15.81:6801","nonce":9680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":9680},{"type":"v1","addr":"172.21.15.81:6803","nonce":9680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":9680},{"type":"v1","addr":"172.21.15.81:6807","nonce":9680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":9680},{"type":"v1","addr":"172.21.15.81:6805","nonce":9680}]},"public_addr":"172.21.15.81:6801/9680","cluster_addr":"172.21.15.81:6803/9680","heartbeat_back_addr":"172.21.15.81:6807/9680","heartbeat_front_addr":"172.21.15.81:6805/9680","state":["exists","up"]},{"osd":7,"uuid":"bb0ada9f-667e-403a-8317-e2ddc59c7dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":9678},{"type":"v1","addr":"172.21.15.81:6809","nonce":9678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":9678},{"type":"v1","addr":"172.21.15.81:6811","nonce":9678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":9678},{"type":"v1","addr":"172.21.15.81:6820","nonce":9678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":9678},{"type":"v1","addr":"172.21.15.81:6815","nonce":9678}]},"public_addr":"172.21.15.81:6809/9678","cluster_addr":"172.21.15.81:6811/9678","heartbeat_back_addr":"172.21.15.81:6820/9678","heartbeat_front_addr":"172.21.15.81:6815/9678","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.455163+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:18.395441+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.217011+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:18.335336+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.237049+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.793779+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.289402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.600264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-09T20:34:29.012 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-09T20:34:24.453766+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'20', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-09T20:34:29.012 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-09T20:34:29.822 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 8 2019-12-09T20:34:29.836 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2019-12-09T20:34:29.836 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2019-12-09T20:34:29.837 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2019-12-09T20:34:30.833 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:34:30.851 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"active_gid":4105,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":9311},{"type":"v1","addr":"172.21.15.67:6801","nonce":9311}]},"active_addr":"172.21.15.67:6801/9311","active_change":"2019-12-09T20:33:56.009419+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"1","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["ceph-daemon-package","root"],"desc":"mode for remote execution of ceph-daemon","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"1","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["ceph-daemon-package","root"],"desc":"mode for remote execution of ceph-daemon","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]}} 2019-12-09T20:34:30.862 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2019-12-09T20:34:30.863 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2019-12-09T20:34:30.863 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-09T20:34:31.780 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:34:31.780 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":21,"fsid":"bb0ccc8e-29f2-48f9-838e-c4ccdeefe22d","created":"2019-12-09T20:33:52.288217+0000","modified":"2019-12-09T20:34:28.954065+0000","last_up_change":"2019-12-09T20:34:21.509278+0000","last_in_change":"2019-12-09T20:34:21.509278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-09T20:34:24.453766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bcb07227-f2ab-47ba-97ac-3302ac851466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":10703},{"type":"v1","addr":"172.21.15.67:6806","nonce":10703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":10703},{"type":"v1","addr":"172.21.15.67:6810","nonce":10703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":10703},{"type":"v1","addr":"172.21.15.67:6817","nonce":10703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":10703},{"type":"v1","addr":"172.21.15.67:6813","nonce":10703}]},"public_addr":"172.21.15.67:6806/10703","cluster_addr":"172.21.15.67:6810/10703","heartbeat_back_addr":"172.21.15.67:6817/10703","heartbeat_front_addr":"172.21.15.67:6813/10703","state":["exists","up"]},{"osd":1,"uuid":"194d5481-21e3-4653-9ca6-8154ace63907","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":10706},{"type":"v1","addr":"172.21.15.67:6827","nonce":10706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":10706},{"type":"v1","addr":"172.21.15.67:6829","nonce":10706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":10706},{"type":"v1","addr":"172.21.15.67:6833","nonce":10706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":10706},{"type":"v1","addr":"172.21.15.67:6831","nonce":10706}]},"public_addr":"172.21.15.67:6827/10706","cluster_addr":"172.21.15.67:6829/10706","heartbeat_back_addr":"172.21.15.67:6833/10706","heartbeat_front_addr":"172.21.15.67:6831/10706","state":["exists","up"]},{"osd":2,"uuid":"ddcb83a9-ab24-4e56-a27c-d75f3b589c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":10705},{"type":"v1","addr":"172.21.15.67:6803","nonce":10705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":10705},{"type":"v1","addr":"172.21.15.67:6807","nonce":10705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":10705},{"type":"v1","addr":"172.21.15.67:6816","nonce":10705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":10705},{"type":"v1","addr":"172.21.15.67:6811","nonce":10705}]},"public_addr":"172.21.15.67:6803/10705","cluster_addr":"172.21.15.67:6807/10705","heartbeat_back_addr":"172.21.15.67:6816/10705","heartbeat_front_addr":"172.21.15.67:6811/10705","state":["exists","up"]},{"osd":3,"uuid":"bde673fa-eeb4-4878-bbde-0ec9ae07d78c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":10704},{"type":"v1","addr":"172.21.15.67:6819","nonce":10704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":10704},{"type":"v1","addr":"172.21.15.67:6821","nonce":10704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":10704},{"type":"v1","addr":"172.21.15.67:6825","nonce":10704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":10704},{"type":"v1","addr":"172.21.15.67:6823","nonce":10704}]},"public_addr":"172.21.15.67:6819/10704","cluster_addr":"172.21.15.67:6821/10704","heartbeat_back_addr":"172.21.15.67:6825/10704","heartbeat_front_addr":"172.21.15.67:6823/10704","state":["exists","up"]},{"osd":4,"uuid":"c5f858b6-a265-49c9-8120-c594db9230c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":9681},{"type":"v1","addr":"172.21.15.81:6818","nonce":9681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":9681},{"type":"v1","addr":"172.21.15.81:6825","nonce":9681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":9681},{"type":"v1","addr":"172.21.15.81:6831","nonce":9681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":9681},{"type":"v1","addr":"172.21.15.81:6829","nonce":9681}]},"public_addr":"172.21.15.81:6818/9681","cluster_addr":"172.21.15.81:6825/9681","heartbeat_back_addr":"172.21.15.81:6831/9681","heartbeat_front_addr":"172.21.15.81:6829/9681","state":["exists","up"]},{"osd":5,"uuid":"e2d52d92-a27b-4582-8713-171ce412b4a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":9679},{"type":"v1","addr":"172.21.15.81:6814","nonce":9679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":9679},{"type":"v1","addr":"172.21.15.81:6821","nonce":9679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":9679},{"type":"v1","addr":"172.21.15.81:6827","nonce":9679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":9679},{"type":"v1","addr":"172.21.15.81:6824","nonce":9679}]},"public_addr":"172.21.15.81:6814/9679","cluster_addr":"172.21.15.81:6821/9679","heartbeat_back_addr":"172.21.15.81:6827/9679","heartbeat_front_addr":"172.21.15.81:6824/9679","state":["exists","up"]},{"osd":6,"uuid":"2f23a442-4fc3-417f-aeca-c0d9130e5f85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":9680},{"type":"v1","addr":"172.21.15.81:6801","nonce":9680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":9680},{"type":"v1","addr":"172.21.15.81:6803","nonce":9680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":9680},{"type":"v1","addr":"172.21.15.81:6807","nonce":9680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":9680},{"type":"v1","addr":"172.21.15.81:6805","nonce":9680}]},"public_addr":"172.21.15.81:6801/9680","cluster_addr":"172.21.15.81:6803/9680","heartbeat_back_addr":"172.21.15.81:6807/9680","heartbeat_front_addr":"172.21.15.81:6805/9680","state":["exists","up"]},{"osd":7,"uuid":"bb0ada9f-667e-403a-8317-e2ddc59c7dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":9678},{"type":"v1","addr":"172.21.15.81:6809","nonce":9678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":9678},{"type":"v1","addr":"172.21.15.81:6811","nonce":9678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":9678},{"type":"v1","addr":"172.21.15.81:6820","nonce":9678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":9678},{"type":"v1","addr":"172.21.15.81:6815","nonce":9678}]},"public_addr":"172.21.15.81:6809/9678","cluster_addr":"172.21.15.81:6811/9678","heartbeat_back_addr":"172.21.15.81:6820/9678","heartbeat_front_addr":"172.21.15.81:6815/9678","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.455163+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:18.395441+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.217011+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:18.335336+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.237049+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.793779+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.289402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.600264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-09T20:34:31.798 INFO:tasks.ceph.ceph_manager.ceph:all up! 2019-12-09T20:34:31.798 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-09T20:34:32.638 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:34:32.638 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":21,"fsid":"bb0ccc8e-29f2-48f9-838e-c4ccdeefe22d","created":"2019-12-09T20:33:52.288217+0000","modified":"2019-12-09T20:34:28.954065+0000","last_up_change":"2019-12-09T20:34:21.509278+0000","last_in_change":"2019-12-09T20:34:21.509278+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-09T20:34:24.453766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"bcb07227-f2ab-47ba-97ac-3302ac851466","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6805","nonce":10703},{"type":"v1","addr":"172.21.15.67:6806","nonce":10703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6809","nonce":10703},{"type":"v1","addr":"172.21.15.67:6810","nonce":10703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":10703},{"type":"v1","addr":"172.21.15.67:6817","nonce":10703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":10703},{"type":"v1","addr":"172.21.15.67:6813","nonce":10703}]},"public_addr":"172.21.15.67:6806/10703","cluster_addr":"172.21.15.67:6810/10703","heartbeat_back_addr":"172.21.15.67:6817/10703","heartbeat_front_addr":"172.21.15.67:6813/10703","state":["exists","up"]},{"osd":1,"uuid":"194d5481-21e3-4653-9ca6-8154ace63907","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":10706},{"type":"v1","addr":"172.21.15.67:6827","nonce":10706}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":10706},{"type":"v1","addr":"172.21.15.67:6829","nonce":10706}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":10706},{"type":"v1","addr":"172.21.15.67:6833","nonce":10706}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":10706},{"type":"v1","addr":"172.21.15.67:6831","nonce":10706}]},"public_addr":"172.21.15.67:6827/10706","cluster_addr":"172.21.15.67:6829/10706","heartbeat_back_addr":"172.21.15.67:6833/10706","heartbeat_front_addr":"172.21.15.67:6831/10706","state":["exists","up"]},{"osd":2,"uuid":"ddcb83a9-ab24-4e56-a27c-d75f3b589c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":10705},{"type":"v1","addr":"172.21.15.67:6803","nonce":10705}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":10705},{"type":"v1","addr":"172.21.15.67:6807","nonce":10705}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":10705},{"type":"v1","addr":"172.21.15.67:6816","nonce":10705}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":10705},{"type":"v1","addr":"172.21.15.67:6811","nonce":10705}]},"public_addr":"172.21.15.67:6803/10705","cluster_addr":"172.21.15.67:6807/10705","heartbeat_back_addr":"172.21.15.67:6816/10705","heartbeat_front_addr":"172.21.15.67:6811/10705","state":["exists","up"]},{"osd":3,"uuid":"bde673fa-eeb4-4878-bbde-0ec9ae07d78c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":10704},{"type":"v1","addr":"172.21.15.67:6819","nonce":10704}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":10704},{"type":"v1","addr":"172.21.15.67:6821","nonce":10704}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":10704},{"type":"v1","addr":"172.21.15.67:6825","nonce":10704}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":10704},{"type":"v1","addr":"172.21.15.67:6823","nonce":10704}]},"public_addr":"172.21.15.67:6819/10704","cluster_addr":"172.21.15.67:6821/10704","heartbeat_back_addr":"172.21.15.67:6825/10704","heartbeat_front_addr":"172.21.15.67:6823/10704","state":["exists","up"]},{"osd":4,"uuid":"c5f858b6-a265-49c9-8120-c594db9230c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6816","nonce":9681},{"type":"v1","addr":"172.21.15.81:6818","nonce":9681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6822","nonce":9681},{"type":"v1","addr":"172.21.15.81:6825","nonce":9681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6830","nonce":9681},{"type":"v1","addr":"172.21.15.81:6831","nonce":9681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6828","nonce":9681},{"type":"v1","addr":"172.21.15.81:6829","nonce":9681}]},"public_addr":"172.21.15.81:6818/9681","cluster_addr":"172.21.15.81:6825/9681","heartbeat_back_addr":"172.21.15.81:6831/9681","heartbeat_front_addr":"172.21.15.81:6829/9681","state":["exists","up"]},{"osd":5,"uuid":"e2d52d92-a27b-4582-8713-171ce412b4a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6812","nonce":9679},{"type":"v1","addr":"172.21.15.81:6814","nonce":9679}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6819","nonce":9679},{"type":"v1","addr":"172.21.15.81:6821","nonce":9679}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6826","nonce":9679},{"type":"v1","addr":"172.21.15.81:6827","nonce":9679}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6823","nonce":9679},{"type":"v1","addr":"172.21.15.81:6824","nonce":9679}]},"public_addr":"172.21.15.81:6814/9679","cluster_addr":"172.21.15.81:6821/9679","heartbeat_back_addr":"172.21.15.81:6827/9679","heartbeat_front_addr":"172.21.15.81:6824/9679","state":["exists","up"]},{"osd":6,"uuid":"2f23a442-4fc3-417f-aeca-c0d9130e5f85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6800","nonce":9680},{"type":"v1","addr":"172.21.15.81:6801","nonce":9680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6802","nonce":9680},{"type":"v1","addr":"172.21.15.81:6803","nonce":9680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6806","nonce":9680},{"type":"v1","addr":"172.21.15.81:6807","nonce":9680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6804","nonce":9680},{"type":"v1","addr":"172.21.15.81:6805","nonce":9680}]},"public_addr":"172.21.15.81:6801/9680","cluster_addr":"172.21.15.81:6803/9680","heartbeat_back_addr":"172.21.15.81:6807/9680","heartbeat_front_addr":"172.21.15.81:6805/9680","state":["exists","up"]},{"osd":7,"uuid":"bb0ada9f-667e-403a-8317-e2ddc59c7dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6808","nonce":9678},{"type":"v1","addr":"172.21.15.81:6809","nonce":9678}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6810","nonce":9678},{"type":"v1","addr":"172.21.15.81:6811","nonce":9678}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6817","nonce":9678},{"type":"v1","addr":"172.21.15.81:6820","nonce":9678}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.81:6813","nonce":9678},{"type":"v1","addr":"172.21.15.81:6815","nonce":9678}]},"public_addr":"172.21.15.81:6809/9678","cluster_addr":"172.21.15.81:6811/9678","heartbeat_back_addr":"172.21.15.81:6820/9678","heartbeat_front_addr":"172.21.15.81:6815/9678","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.455163+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:18.395441+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.217011+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:18.335336+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.237049+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.793779+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.289402+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-09T20:34:17.600264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-09T20:34:32.653 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2019-12-09T20:34:33.081 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2019-12-09T20:34:33.081 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2019-12-09T20:34:33.605 INFO:teuthology.orchestra.run.smithi067.stdout:64424509445 2019-12-09T20:34:33.606 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2019-12-09T20:34:34.123 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2019-12-09T20:34:34.124 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2019-12-09T20:34:34.575 INFO:teuthology.orchestra.run.smithi067.stdout:64424509445 2019-12-09T20:34:34.575 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2019-12-09T20:34:35.201 INFO:teuthology.orchestra.run.smithi067.stdout:60129542150 2019-12-09T20:34:35.202 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2019-12-09T20:34:35.861 INFO:teuthology.orchestra.run.smithi067.stdout:64424509446 2019-12-09T20:34:35.862 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2019-12-09T20:34:36.536 INFO:teuthology.orchestra.run.smithi067.stdout:60129542150 2019-12-09T20:34:36.536 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2019-12-09T20:34:37.110 INFO:teuthology.orchestra.run.smithi067.stdout:55834574853 2019-12-09T20:34:37.111 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-12-09T20:34:38.014 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2019-12-09T20:34:38.028 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542149 for osd.0 2019-12-09T20:34:38.028 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-12-09T20:34:38.792 INFO:teuthology.orchestra.run.smithi067.stdout:64424509445 2019-12-09T20:34:38.806 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.1 2019-12-09T20:34:38.807 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2019-12-09T20:34:39.628 INFO:teuthology.orchestra.run.smithi067.stdout:60129542150 2019-12-09T20:34:39.641 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.2 2019-12-09T20:34:39.642 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2019-12-09T20:34:40.440 INFO:teuthology.orchestra.run.smithi067.stdout:64424509446 2019-12-09T20:34:40.456 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509446 for osd.3 2019-12-09T20:34:40.457 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-12-09T20:34:41.356 INFO:teuthology.orchestra.run.smithi067.stdout:60129542151 2019-12-09T20:34:41.369 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.4 2019-12-09T20:34:41.370 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2019-12-09T20:34:42.060 INFO:teuthology.orchestra.run.smithi067.stdout:64424509447 2019-12-09T20:34:42.075 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509447 for osd.5 2019-12-09T20:34:42.076 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2019-12-09T20:34:42.965 INFO:teuthology.orchestra.run.smithi067.stdout:60129542151 2019-12-09T20:34:42.984 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.6 2019-12-09T20:34:42.985 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2019-12-09T20:34:43.762 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2019-12-09T20:34:43.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.7 2019-12-09T20:34:43.776 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2019-12-09T20:34:43.776 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-09T20:34:44.522 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2019-12-09T20:34:44.522 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:34:44.543 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2019-12-09T20:34:44.424213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390304,"kb_used_data":1632,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584416,"statfs":{"total":773094113280,"available":764502441984,"internally_reserved":8589934592,"allocated":1671168,"data_stored":540720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.137711+0000","last_change":"2019-12-09T20:34:27.608011+0000","last_active":"2019-12-09T20:34:28.137711+0000","last_peered":"2019-12-09T20:34:28.137711+0000","last_clean":"2019-12-09T20:34:28.137711+0000","last_became_active":"2019-12-09T20:34:27.543112+0000","last_became_peered":"2019-12-09T20:34:27.543112+0000","last_unstale":"2019-12-09T20:34:28.137711+0000","last_undegraded":"2019-12-09T20:34:28.137711+0000","last_fullsized":"2019-12-09T20:34:28.137711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.133657+0000","last_change":"2019-12-09T20:34:27.662429+0000","last_active":"2019-12-09T20:34:28.133657+0000","last_peered":"2019-12-09T20:34:28.133657+0000","last_clean":"2019-12-09T20:34:28.133657+0000","last_became_active":"2019-12-09T20:34:27.648753+0000","last_became_peered":"2019-12-09T20:34:27.648753+0000","last_unstale":"2019-12-09T20:34:28.133657+0000","last_undegraded":"2019-12-09T20:34:28.133657+0000","last_fullsized":"2019-12-09T20:34:28.133657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.912870+0000","last_change":"2019-12-09T20:34:28.887949+0000","last_active":"2019-12-09T20:34:28.912870+0000","last_peered":"2019-12-09T20:34:28.912870+0000","last_clean":"2019-12-09T20:34:28.912870+0000","last_became_active":"2019-12-09T20:34:28.825459+0000","last_became_peered":"2019-12-09T20:34:28.825459+0000","last_unstale":"2019-12-09T20:34:28.912870+0000","last_undegraded":"2019-12-09T20:34:28.912870+0000","last_fullsized":"2019-12-09T20:34:28.912870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.452891+0000","last_change":"2019-12-09T20:34:28.427032+0000","last_active":"2019-12-09T20:34:28.452891+0000","last_peered":"2019-12-09T20:34:28.452891+0000","last_clean":"2019-12-09T20:34:28.452891+0000","last_became_active":"2019-12-09T20:34:28.397294+0000","last_became_peered":"2019-12-09T20:34:28.397294+0000","last_unstale":"2019-12-09T20:34:28.452891+0000","last_undegraded":"2019-12-09T20:34:28.452891+0000","last_fullsized":"2019-12-09T20:34:28.452891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.969509+0000","last_change":"2019-12-09T20:34:28.933966+0000","last_active":"2019-12-09T20:34:28.969509+0000","last_peered":"2019-12-09T20:34:28.969509+0000","last_clean":"2019-12-09T20:34:28.969509+0000","last_became_active":"2019-12-09T20:34:28.884995+0000","last_became_peered":"2019-12-09T20:34:28.884995+0000","last_unstale":"2019-12-09T20:34:28.969509+0000","last_undegraded":"2019-12-09T20:34:28.969509+0000","last_fullsized":"2019-12-09T20:34:28.969509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.922931+0000","last_change":"2019-12-09T20:34:28.894795+0000","last_active":"2019-12-09T20:34:28.922931+0000","last_peered":"2019-12-09T20:34:28.922931+0000","last_clean":"2019-12-09T20:34:28.922931+0000","last_became_active":"2019-12-09T20:34:28.829816+0000","last_became_peered":"2019-12-09T20:34:28.829816+0000","last_unstale":"2019-12-09T20:34:28.922931+0000","last_undegraded":"2019-12-09T20:34:28.922931+0000","last_fullsized":"2019-12-09T20:34:28.922931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.963882+0000","last_change":"2019-12-09T20:34:28.914604+0000","last_active":"2019-12-09T20:34:28.963882+0000","last_peered":"2019-12-09T20:34:28.963882+0000","last_clean":"2019-12-09T20:34:28.963882+0000","last_became_active":"2019-12-09T20:34:28.854995+0000","last_became_peered":"2019-12-09T20:34:28.854995+0000","last_unstale":"2019-12-09T20:34:28.963882+0000","last_undegraded":"2019-12-09T20:34:28.963882+0000","last_fullsized":"2019-12-09T20:34:28.963882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.394770+0000","last_change":"2019-12-09T20:34:28.368023+0000","last_active":"2019-12-09T20:34:28.394770+0000","last_peered":"2019-12-09T20:34:28.394770+0000","last_clean":"2019-12-09T20:34:28.394770+0000","last_became_active":"2019-12-09T20:34:28.322892+0000","last_became_peered":"2019-12-09T20:34:28.322892+0000","last_unstale":"2019-12-09T20:34:28.394770+0000","last_undegraded":"2019-12-09T20:34:28.394770+0000","last_fullsized":"2019-12-09T20:34:28.394770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":13.700}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.426}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.373}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.902}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":34.552}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":38.984}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":23.864}]}]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":18.579}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.889}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.879}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.596}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":29.460}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.287}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.618}]}]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":12.457}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.191}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":22.064}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.061}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.299}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":23.055}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.865}]}]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":22.659}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":37.234}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.069}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.742}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.384}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":30.471}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":21.215}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":33.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":27.981}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.715}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":13.511}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":36.778}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":24.853}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":23.452}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.291}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.090}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":22.987}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":30.191}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":25.899}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":34.430}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.110}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.438}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":21.562}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":29.757}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":34.252}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.682}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":35.833}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.299}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.737}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":33.536}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":29.975}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":28.052}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":34.349}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":40.467}]}]}]}} 2019-12-09T20:34:44.696 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-09T20:34:45.630 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2019-12-09T20:34:45.630 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:34:45.643 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2019-12-09T20:34:44.424213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390304,"kb_used_data":1632,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584416,"statfs":{"total":773094113280,"available":764502441984,"internally_reserved":8589934592,"allocated":1671168,"data_stored":540720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.137711+0000","last_change":"2019-12-09T20:34:27.608011+0000","last_active":"2019-12-09T20:34:28.137711+0000","last_peered":"2019-12-09T20:34:28.137711+0000","last_clean":"2019-12-09T20:34:28.137711+0000","last_became_active":"2019-12-09T20:34:27.543112+0000","last_became_peered":"2019-12-09T20:34:27.543112+0000","last_unstale":"2019-12-09T20:34:28.137711+0000","last_undegraded":"2019-12-09T20:34:28.137711+0000","last_fullsized":"2019-12-09T20:34:28.137711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.133657+0000","last_change":"2019-12-09T20:34:27.662429+0000","last_active":"2019-12-09T20:34:28.133657+0000","last_peered":"2019-12-09T20:34:28.133657+0000","last_clean":"2019-12-09T20:34:28.133657+0000","last_became_active":"2019-12-09T20:34:27.648753+0000","last_became_peered":"2019-12-09T20:34:27.648753+0000","last_unstale":"2019-12-09T20:34:28.133657+0000","last_undegraded":"2019-12-09T20:34:28.133657+0000","last_fullsized":"2019-12-09T20:34:28.133657+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.912870+0000","last_change":"2019-12-09T20:34:28.887949+0000","last_active":"2019-12-09T20:34:28.912870+0000","last_peered":"2019-12-09T20:34:28.912870+0000","last_clean":"2019-12-09T20:34:28.912870+0000","last_became_active":"2019-12-09T20:34:28.825459+0000","last_became_peered":"2019-12-09T20:34:28.825459+0000","last_unstale":"2019-12-09T20:34:28.912870+0000","last_undegraded":"2019-12-09T20:34:28.912870+0000","last_fullsized":"2019-12-09T20:34:28.912870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.452891+0000","last_change":"2019-12-09T20:34:28.427032+0000","last_active":"2019-12-09T20:34:28.452891+0000","last_peered":"2019-12-09T20:34:28.452891+0000","last_clean":"2019-12-09T20:34:28.452891+0000","last_became_active":"2019-12-09T20:34:28.397294+0000","last_became_peered":"2019-12-09T20:34:28.397294+0000","last_unstale":"2019-12-09T20:34:28.452891+0000","last_undegraded":"2019-12-09T20:34:28.452891+0000","last_fullsized":"2019-12-09T20:34:28.452891+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.969509+0000","last_change":"2019-12-09T20:34:28.933966+0000","last_active":"2019-12-09T20:34:28.969509+0000","last_peered":"2019-12-09T20:34:28.969509+0000","last_clean":"2019-12-09T20:34:28.969509+0000","last_became_active":"2019-12-09T20:34:28.884995+0000","last_became_peered":"2019-12-09T20:34:28.884995+0000","last_unstale":"2019-12-09T20:34:28.969509+0000","last_undegraded":"2019-12-09T20:34:28.969509+0000","last_fullsized":"2019-12-09T20:34:28.969509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.922931+0000","last_change":"2019-12-09T20:34:28.894795+0000","last_active":"2019-12-09T20:34:28.922931+0000","last_peered":"2019-12-09T20:34:28.922931+0000","last_clean":"2019-12-09T20:34:28.922931+0000","last_became_active":"2019-12-09T20:34:28.829816+0000","last_became_peered":"2019-12-09T20:34:28.829816+0000","last_unstale":"2019-12-09T20:34:28.922931+0000","last_undegraded":"2019-12-09T20:34:28.922931+0000","last_fullsized":"2019-12-09T20:34:28.922931+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.963882+0000","last_change":"2019-12-09T20:34:28.914604+0000","last_active":"2019-12-09T20:34:28.963882+0000","last_peered":"2019-12-09T20:34:28.963882+0000","last_clean":"2019-12-09T20:34:28.963882+0000","last_became_active":"2019-12-09T20:34:28.854995+0000","last_became_peered":"2019-12-09T20:34:28.854995+0000","last_unstale":"2019-12-09T20:34:28.963882+0000","last_undegraded":"2019-12-09T20:34:28.963882+0000","last_fullsized":"2019-12-09T20:34:28.963882+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2019-12-09T20:34:28.394770+0000","last_change":"2019-12-09T20:34:28.368023+0000","last_active":"2019-12-09T20:34:28.394770+0000","last_peered":"2019-12-09T20:34:28.394770+0000","last_clean":"2019-12-09T20:34:28.394770+0000","last_became_active":"2019-12-09T20:34:28.322892+0000","last_became_peered":"2019-12-09T20:34:28.322892+0000","last_unstale":"2019-12-09T20:34:28.394770+0000","last_undegraded":"2019-12-09T20:34:28.394770+0000","last_fullsized":"2019-12-09T20:34:28.394770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-09T20:34:25.291929+0000","last_clean_scrub_stamp":"2019-12-09T20:34:25.291929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":5,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":13.700}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.426}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.373}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.902}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":34.552}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":38.984}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":23.864}]}]},{"osd":3,"up_from":15,"seq":64424509447,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":18.579}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.889}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.879}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.596}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":29.460}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.287}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.618}]}]},{"osd":1,"up_from":15,"seq":64424509447,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":12.457}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":14.191}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":22.064}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.061}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.299}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":23.055}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.865}]}]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":22.659}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":37.234}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.069}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.742}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":26.384}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":30.471}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":21.215}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":33.501}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":27.981}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.715}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":13.511}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":36.778}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":24.853}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":23.452}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.291}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.090}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":22.987}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":30.191}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":25.899}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":34.430}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":20.110}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":19.438}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":15.748}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":21.562}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":29.757}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":34.252}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":32.682}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":35.833}]}]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048788,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323052,"statfs":{"total":96636764160,"available":95562805248,"internally_reserved":1073741824,"allocated":208896,"data_stored":67590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":16.299}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":31.737}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":33.536}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":29.975}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":28.052}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":34.349}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":40.467}]}]}]}} 2019-12-09T20:34:45.645 INFO:tasks.ceph.ceph_manager.ceph:clean! 2019-12-09T20:34:45.645 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2019-12-09T20:34:45.646 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2019-12-09T20:34:45.646 INFO:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2019-12-09T20:34:46.377 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:34:46.393 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2019-12-09T20:34:46.394 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2019-12-09T20:34:46.394 INFO:teuthology.run_tasks:Running task openssl_keys... 2019-12-09T20:34:46.410 INFO:teuthology.run_tasks:Running task rgw... 2019-12-09T20:34:46.430 DEBUG:tasks.rgw:config is {'client.0': {'valgrind': ['--tool=memcheck', '--max-threads=1024']}} 2019-12-09T20:34:46.430 DEBUG:tasks.rgw:client list is ['client.0'] 2019-12-09T20:34:46.430 INFO:tasks.rgw:Creating data pools 2019-12-09T20:34:46.431 DEBUG:tasks.rgw:Obtaining remote for client client.0 2019-12-09T20:34:46.431 INFO:teuthology.orchestra.run.smithi067:> sudo ceph osd erasure-code-profile set client.0 crush-failure-domain=osd k=3 m=1 name=testprofile --cluster ceph 2019-12-09T20:34:47.953 INFO:teuthology.orchestra.run.smithi067:> sudo ceph osd pool create default.rgw.buckets.data 64 64 erasure client.0 --cluster ceph 2019-12-09T20:34:50.167 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'default.rgw.buckets.data' created 2019-12-09T20:34:50.183 INFO:teuthology.orchestra.run.smithi067:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2019-12-09T20:34:51.645 INFO:teuthology.orchestra.run.smithi067.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2019-12-09T20:34:51.664 INFO:teuthology.orchestra.run.smithi067:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2019-12-09T20:34:52.793 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'default.rgw.buckets.index' created 2019-12-09T20:34:52.811 INFO:teuthology.orchestra.run.smithi067:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2019-12-09T20:34:54.919 INFO:teuthology.orchestra.run.smithi067.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2019-12-09T20:34:54.935 DEBUG:tasks.rgw:Pools created 2019-12-09T20:34:54.936 INFO:tasks.rgw:Configuring compression type = random 2019-12-09T20:34:54.936 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2019-12-09T20:34:54.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2019-12-09T20:34:54.936 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2019-12-09T20:34:55.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:34:55.064+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:34:57.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:34:57.704+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:34:57.723 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:34:57.704+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:34:58.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:34:58.621+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:34:58.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:34:58.621+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:34:58.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:34:58.853+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:34:58.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:34:58.853+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:34:59.003 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:34:58.985+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:00.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.424+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.432+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:00.451 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.432+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:00.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.437+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:00.456 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.437+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:00.463 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.445+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.445+0000 7fdb66fc4ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T20:35:00.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.445+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:00.472 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.454+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:00.473 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.454+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:00.479 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.460+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.460+0000 7fdb66fc4ec0 10 Creating default zonegroup 2019-12-09T20:35:00.480 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.460+0000 7fdb66fc4ec0 20 get_system_obj_state: rctx=0x7fffe823b2c0 obj=:.pools.avail state=0x555c1a5c1b50 s->prefetch_data=0 2019-12-09T20:35:00.481 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.463+0000 7fdb66fc4ec0 10 couldn't find old data placement pools config, setting up new ones for the zone 2019-12-09T20:35:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.604+0000 7fdb66fc4ec0 10 WARNING: store->list_zones() returned r=-2 2019-12-09T20:35:00.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.604+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:00.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:00.609+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:01.695 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:01.673+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:01.684+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:01.708 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:01.684+0000 7fdb66fc4ec0 10 could not read realm id: (2) No such file or directory 2019-12-09T20:35:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:01.684+0000 7fdb66fc4ec0 10 WARNING: failed to set zone as default, r=-22 2019-12-09T20:35:01.709 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:01.684+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:01.714 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:01.696+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:02.378 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.360+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:02.444 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.426+0000 7fdb66fc4ec0 20 rados_obj.operate() r=0 bl.length=805 2019-12-09T20:35:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.426+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:02.521 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.502+0000 7fdb541ee700 0 -- 172.21.15.67:0/3636707132 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x555c1a5bebc0 msgr2=0x555c1a5c0fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:35:02.521 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.502+0000 7fdb66fc4ec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-09T20:35:02.521 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.502+0000 7fdb66fc4ec0 20 zonegroup default 2019-12-09T20:35:02.521 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.502+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:02.542 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.523+0000 7fdb66fc4ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:02.542 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.524+0000 7fdb66fc4ec0 10 Cannot find current period zone using local zone 2019-12-09T20:35:02.542 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.524+0000 7fdb66fc4ec0 10 Using default name default 2019-12-09T20:35:02.543 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.524+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.572+0000 7fdb66fc4ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:02.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.572+0000 7fdb66fc4ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:02.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.586+0000 7fdb66fc4ec0 20 rados_obj.operate() r=0 bl.length=805 2019-12-09T20:35:02.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.586+0000 7fdb66fc4ec0 20 zone default found 2019-12-09T20:35:02.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.586+0000 7fdb66fc4ec0 20 started sync module instance, tier type = 2019-12-09T20:35:02.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:02.586+0000 7fdb66fc4ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T20:35:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:06.210+0000 7fdb549ef700 0 -- 172.21.15.67:0/3636707132 >> [v2:172.21.15.67:6802/10705,v1:172.21.15.67:6803/10705] conn(0x555c1a5c6460 msgr2=0x555c1a5c8880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:35:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.677+0000 7fdb66fc4ec0 20 add_watcher() i=0 2019-12-09T20:35:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.677+0000 7fdb66fc4ec0 20 add_watcher() i=1 2019-12-09T20:35:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.677+0000 7fdb66fc4ec0 20 add_watcher() i=2 2019-12-09T20:35:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.677+0000 7fdb66fc4ec0 20 add_watcher() i=3 2019-12-09T20:35:07.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.677+0000 7fdb66fc4ec0 20 add_watcher() i=4 2019-12-09T20:35:07.698 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.677+0000 7fdb66fc4ec0 20 add_watcher() i=5 2019-12-09T20:35:07.752 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.731+0000 7fdb66fc4ec0 20 add_watcher() i=6 2019-12-09T20:35:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.787+0000 7fdb66fc4ec0 20 add_watcher() i=7 2019-12-09T20:35:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.787+0000 7fdb66fc4ec0 2 all 8 watchers are set, enabling cache 2019-12-09T20:35:07.807 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:07.788+0000 7fdb24fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:35:11.750 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:11.732+0000 7fdb66fc4ec0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2019-12-09T20:35:16.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:16.010+0000 7fdb66fc4ec0 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.log 2019-12-09T20:35:20.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:20.770+0000 7fda92ffd700 20 reqs_thread_entry: start 2019-12-09T20:35:20.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:20.770+0000 7fda90ff9700 20 reqs_thread_entry: start 2019-12-09T20:35:20.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:20.770+0000 7fdb66fc4ec0 20 init_complete bucket index max shards: 0 2019-12-09T20:35:20.790 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:20.771+0000 7fda7effd700 20 reqs_thread_entry: start 2019-12-09T20:35:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.170+0000 7fdb66fc4ec0 20 remove_watcher() i=0 2019-12-09T20:35:21.190 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.170+0000 7fdb66fc4ec0 2 removed watcher, disabling cache 2019-12-09T20:35:21.278 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.259+0000 7fdb66fc4ec0 20 remove_watcher() i=1 2019-12-09T20:35:21.348 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.329+0000 7fdb66fc4ec0 20 remove_watcher() i=2 2019-12-09T20:35:21.446 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.427+0000 7fdb66fc4ec0 20 remove_watcher() i=3 2019-12-09T20:35:21.555 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.535+0000 7fdb66fc4ec0 20 remove_watcher() i=4 2019-12-09T20:35:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.593+0000 7fdb66fc4ec0 20 remove_watcher() i=5 2019-12-09T20:35:21.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.672+0000 7fdb66fc4ec0 20 remove_watcher() i=6 2019-12-09T20:35:21.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.719+0000 7fdb66fc4ec0 20 remove_watcher() i=7 2019-12-09T20:35:21.748 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2019-12-09T20:35:21.748 DEBUG:tasks.util.rgw: json result: [] 2019-12-09T20:35:21.748 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2019-12-09T20:35:21.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2019-12-09T20:35:21.749 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random 2019-12-09T20:35:21.987 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.968+0000 7f7433fff700 20 reqs_thread_entry: start 2019-12-09T20:35:21.987 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.968+0000 7f74ff1f2ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.004 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.985+0000 7f74ff1f2ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:22.004 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:21.985+0000 7f74ff1f2ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.000+0000 7f74ff1f2ec0 20 rados_obj.operate() r=0 bl.length=805 2019-12-09T20:35:22.019 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.000+0000 7f74ff1f2ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.006+0000 7f74ff1f2ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:22.025 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.006+0000 7f74ff1f2ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.015+0000 7f74ff1f2ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:22.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.015+0000 7f74ff1f2ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.047 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.028+0000 7f74ff1f2ec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-09T20:35:22.226 INFO:teuthology.orchestra.run.smithi067.stdout:{"id":"1ad16726-ce4c-4982-bd45-b63e0fe58407","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2019-12-09T20:35:22.226 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'roles_pool': u'default.rgw.meta:roles', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'user_email_pool': u'default.rgw.meta:users.email', u'realm_id': u'', u'placement_pools': [{u'val': {u'storage_classes': {u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'id': u'1ad16726-ce4c-4982-bd45-b63e0fe58407', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2019-12-09T20:35:22.226 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2019-12-09T20:35:22.227 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2019-12-09T20:35:22.227 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2019-12-09T20:35:22.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.365+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.411 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.392+0000 7fd191ddaec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:22.411 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.392+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.422 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.403+0000 7fd191ddaec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:22.422 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.403+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.428 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.409+0000 7fd191ddaec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:22.428 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.409+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.438 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.419+0000 7fd191ddaec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:22.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.648+0000 7fd191ddaec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T20:35:22.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.648+0000 7fd191ddaec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T20:35:22.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.648+0000 7fd191ddaec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T20:35:22.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.648+0000 7fd191ddaec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T20:35:22.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.648+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.657+0000 7fd191ddaec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:22.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.657+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.667+0000 7fd191ddaec0 20 rados_obj.operate() r=0 bl.length=821 2019-12-09T20:35:22.686 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.667+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.675+0000 7fd191ddaec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:22.694 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.675+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.702 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.682+0000 7fd191ddaec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-09T20:35:22.702 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.682+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.688+0000 7fd191ddaec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.688+0000 7fd191ddaec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T20:35:22.707 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.688+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.714 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.695+0000 7fd191ddaec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:22.715 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.695+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.703+0000 7fd191ddaec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:22.723 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.703+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.732 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.712+0000 7fd191ddaec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-09T20:35:22.732 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.712+0000 7fd191ddaec0 20 zonegroup default 2019-12-09T20:35:22.732 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.712+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.719+0000 7fd191ddaec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:22.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.719+0000 7fd191ddaec0 10 Cannot find current period zone using local zone 2019-12-09T20:35:22.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.719+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.745 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.726+0000 7fd191ddaec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:22.745 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.726+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.754 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.734+0000 7fd191ddaec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:22.754 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.734+0000 7fd191ddaec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.748+0000 7fd191ddaec0 20 rados_obj.operate() r=0 bl.length=821 2019-12-09T20:35:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.748+0000 7fd191ddaec0 20 zone default found 2019-12-09T20:35:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.748+0000 7fd191ddaec0 20 started sync module instance, tier type = 2019-12-09T20:35:22.768 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:22.748+0000 7fd191ddaec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T20:35:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.205+0000 7fd191ddaec0 20 add_watcher() i=0 2019-12-09T20:35:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.206+0000 7fd191ddaec0 20 add_watcher() i=1 2019-12-09T20:35:23.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.206+0000 7fd191ddaec0 20 add_watcher() i=2 2019-12-09T20:35:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.206+0000 7fd191ddaec0 20 add_watcher() i=3 2019-12-09T20:35:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.206+0000 7fd191ddaec0 20 add_watcher() i=4 2019-12-09T20:35:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.206+0000 7fd191ddaec0 20 add_watcher() i=5 2019-12-09T20:35:23.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.206+0000 7fd191ddaec0 20 add_watcher() i=6 2019-12-09T20:35:23.266 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.248+0000 7fd191ddaec0 20 add_watcher() i=7 2019-12-09T20:35:23.267 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.248+0000 7fd191ddaec0 2 all 8 watchers are set, enabling cache 2019-12-09T20:35:23.267 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.248+0000 7fd14f7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:35:23.271 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.252+0000 7fd17f805700 0 -- 172.21.15.67:0/603726116 >> v2:172.21.15.67:3300/0 conn(0x557f4da68360 msgr2=0x557f4da68770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T20:35:23.307 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:23.287+0000 7fd17f805700 0 -- 172.21.15.67:0/3537391846 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557f4da6e940 msgr2=0x557f4da6ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-09T20:35:25.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:25.646+0000 7fd0bdffb700 20 reqs_thread_entry: start 2019-12-09T20:35:25.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:25.646+0000 7fd0b77fe700 20 reqs_thread_entry: start 2019-12-09T20:35:25.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:25.646+0000 7fd191ddaec0 20 init_complete bucket index max shards: 0 2019-12-09T20:35:25.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:25.647+0000 7fd0b5ffb700 20 reqs_thread_entry: start 2019-12-09T20:35:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:25.830+0000 7fd191ddaec0 20 remove_watcher() i=0 2019-12-09T20:35:25.851 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:25.830+0000 7fd191ddaec0 2 removed watcher, disabling cache 2019-12-09T20:35:25.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:25.949+0000 7fd191ddaec0 20 remove_watcher() i=1 2019-12-09T20:35:26.022 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.002+0000 7fd191ddaec0 20 remove_watcher() i=2 2019-12-09T20:35:26.089 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.064+0000 7fd191ddaec0 20 remove_watcher() i=3 2019-12-09T20:35:26.161 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.141+0000 7fd191ddaec0 20 remove_watcher() i=4 2019-12-09T20:35:26.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.193+0000 7fd191ddaec0 20 remove_watcher() i=5 2019-12-09T20:35:26.271 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.251+0000 7fd191ddaec0 20 remove_watcher() i=6 2019-12-09T20:35:26.329 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.309+0000 7fd191ddaec0 20 remove_watcher() i=7 2019-12-09T20:35:26.342 INFO:teuthology.orchestra.run.smithi067.stdout:[] 2019-12-09T20:35:26.343 DEBUG:tasks.util.rgw: json result: [] 2019-12-09T20:35:26.343 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2019-12-09T20:35:26.343 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2019-12-09T20:35:26.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2019-12-09T20:35:26.344 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2019-12-09T20:35:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.592+0000 7fdbbbfff700 20 reqs_thread_entry: start 2019-12-09T20:35:26.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.592+0000 7fdc873f8ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:26.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.602+0000 7fdc873f8ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:26.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.602+0000 7fdc873f8ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:26.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.611+0000 7fdc873f8ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:26.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.611+0000 7fdc873f8ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:26.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.624+0000 7fdc873f8ec0 20 rados_obj.operate() r=0 bl.length=348 2019-12-09T20:35:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.624+0000 7fdc873f8ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:26.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:26.636+0000 7fdc873f8ec0 20 rados_obj.operate() r=0 bl.length=821 2019-12-09T20:35:26.812 INFO:teuthology.orchestra.run.smithi067.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["LUKEWARM","STANDARD"]}}] 2019-12-09T20:35:26.812 DEBUG:tasks.util.rgw: json result: [{u'val': {u'storage_classes': [u'LUKEWARM', u'STANDARD'], u'name': u'default-placement', u'tags': []}, u'key': u'default-placement'}] 2019-12-09T20:35:26.812 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2019-12-09T20:35:26.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2019-12-09T20:35:26.813 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2019-12-09T20:35:27.042 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.022+0000 7f5cd97fa700 20 reqs_thread_entry: start 2019-12-09T20:35:27.043 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.022+0000 7f5d34125ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.058 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.038+0000 7f5d34125ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:27.058 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.038+0000 7f5d34125ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.072 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.051+0000 7f5d34125ec0 20 rados_obj.operate() r=0 bl.length=821 2019-12-09T20:35:27.072 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.051+0000 7f5d34125ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.077 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.056+0000 7f5d34125ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:27.077 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.056+0000 7f5d34125ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.065+0000 7f5d34125ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:27.085 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.065+0000 7f5d34125ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.098 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.078+0000 7f5d34125ec0 20 rados_obj.operate() r=0 bl.length=372 2019-12-09T20:35:27.154 INFO:teuthology.orchestra.run.smithi067.stdout:{"id":"1ad16726-ce4c-4982-bd45-b63e0fe58407","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2019-12-09T20:35:27.154 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'roles_pool': u'default.rgw.meta:roles', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'user_email_pool': u'default.rgw.meta:users.email', u'realm_id': u'', u'placement_pools': [{u'val': {u'storage_classes': {u'LUKEWARM': {u'data_pool': u'default.rgw.buckets.data.lukewarm'}, u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'id': u'1ad16726-ce4c-4982-bd45-b63e0fe58407', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2019-12-09T20:35:27.155 INFO:tasks.rgw:Configuring storage class type = FROZEN 2019-12-09T20:35:27.155 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2019-12-09T20:35:27.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2019-12-09T20:35:27.156 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2019-12-09T20:35:27.337 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.317+0000 7f2452ffd700 20 reqs_thread_entry: start 2019-12-09T20:35:27.337 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.317+0000 7f251d8b5ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.347 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.326+0000 7f251d8b5ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:27.347 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.326+0000 7f251d8b5ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.354 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.334+0000 7f251d8b5ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:27.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.334+0000 7f251d8b5ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.381 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.346+0000 7f251d8b5ec0 20 rados_obj.operate() r=0 bl.length=372 2019-12-09T20:35:27.382 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.346+0000 7f251d8b5ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.382 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.359+0000 7f251d8b5ec0 20 rados_obj.operate() r=0 bl.length=888 2019-12-09T20:35:27.457 INFO:teuthology.orchestra.run.smithi067.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2019-12-09T20:35:27.461 DEBUG:tasks.util.rgw: json result: [{u'val': {u'storage_classes': [u'FROZEN', u'LUKEWARM', u'STANDARD'], u'name': u'default-placement', u'tags': []}, u'key': u'default-placement'}] 2019-12-09T20:35:27.461 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2019-12-09T20:35:27.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2019-12-09T20:35:27.462 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2019-12-09T20:35:27.696 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.675+0000 7f02857fa700 20 reqs_thread_entry: start 2019-12-09T20:35:27.696 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.675+0000 7f033ffb8ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.713 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.693+0000 7f033ffb8ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:27.713 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.693+0000 7f033ffb8ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.729 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.709+0000 7f033ffb8ec0 20 rados_obj.operate() r=0 bl.length=888 2019-12-09T20:35:27.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.709+0000 7f033ffb8ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.715+0000 7f033ffb8ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:35:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.715+0000 7f033ffb8ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.725+0000 7f033ffb8ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:35:27.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.725+0000 7f033ffb8ec0 20 rados->read ofs=0 len=0 2019-12-09T20:35:27.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:35:27.740+0000 7f033ffb8ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:35:27.834 INFO:teuthology.orchestra.run.smithi067.stdout:{"id":"1ad16726-ce4c-4982-bd45-b63e0fe58407","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2019-12-09T20:35:27.834 DEBUG:tasks.util.rgw: json result: {u'control_pool': u'default.rgw.control', u'otp_pool': u'default.rgw.otp', u'usage_log_pool': u'default.rgw.log:usage', u'name': u'default', u'user_uid_pool': u'default.rgw.meta:users.uid', u'gc_pool': u'default.rgw.log:gc', u'intent_log_pool': u'default.rgw.log:intent', u'roles_pool': u'default.rgw.meta:roles', u'lc_pool': u'default.rgw.log:lc', u'user_keys_pool': u'default.rgw.meta:users.keys', u'domain_root': u'default.rgw.meta:root', u'system_key': {u'access_key': u'', u'secret_key': u''}, u'reshard_pool': u'default.rgw.log:reshard', u'log_pool': u'default.rgw.log', u'user_email_pool': u'default.rgw.meta:users.email', u'realm_id': u'', u'placement_pools': [{u'val': {u'storage_classes': {u'FROZEN': {u'data_pool': u'default.rgw.buckets.data.frozen'}, u'LUKEWARM': {u'data_pool': u'default.rgw.buckets.data.lukewarm'}, u'STANDARD': {u'data_pool': u'default.rgw.buckets.data', u'compression_type': u'random'}}, u'data_extra_pool': u'default.rgw.buckets.non-ec', u'index_type': 0, u'index_pool': u'default.rgw.buckets.index'}, u'key': u'default-placement'}], u'id': u'1ad16726-ce4c-4982-bd45-b63e0fe58407', u'user_swift_pool': u'default.rgw.meta:users.swift'} 2019-12-09T20:35:27.834 INFO:tasks.rgw:Starting rgw... 2019-12-09T20:35:27.835 INFO:tasks.rgw:rgw client.0 config is {'valgrind': ['--tool=memcheck', '--max-threads=1024']} 2019-12-09T20:35:27.835 INFO:tasks.rgw:Using civetweb as radosgw frontend 2019-12-09T20:35:27.835 DEBUG:teuthology.misc:running ceph.client.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/ceph.client.0.log', '--time-stamp=yes', '--tool=memcheck', '--max-threads=1024'] 2019-12-09T20:35:27.835 INFO:tasks.rgw.client.0:Restarting daemon 2019-12-09T20:35:27.835 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/ceph.client.0.log --time-stamp=yes --tool=memcheck --max-threads=1024 radosgw --rgw-frontends 'civetweb port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2019-12-09T20:35:27.841 INFO:tasks.rgw.client.0:Started 2019-12-09T20:35:27.841 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi067.front.sepia.ceph.com:80/ 2019-12-09T20:35:27.841 INFO:teuthology.orchestra.run.smithi067:> curl http://smithi067.front.sepia.ceph.com:80/ 2019-12-09T20:35:27.920 INFO:teuthology.orchestra.run.smithi067.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-09T20:35:27.920 INFO:teuthology.orchestra.run.smithi067.stderr: Dload Upload Total Spent Left Speed 2019-12-09T20:35:27.924 INFO:teuthology.orchestra.run.smithi067.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi067.front.sepia.ceph.com:80; Connection refused 2019-12-09T20:35:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 7 2019-12-09T20:35:28.926 INFO:teuthology.orchestra.run.smithi067:> curl http://smithi067.front.sepia.ceph.com:80/ 2019-12-09T20:35:29.012 INFO:teuthology.orchestra.run.smithi067.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-09T20:35:29.012 INFO:teuthology.orchestra.run.smithi067.stderr: Dload Upload Total Spent Left Speed 2019-12-09T20:35:29.015 INFO:teuthology.orchestra.run.smithi067.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi067.front.sepia.ceph.com:80; Connection refused 2019-12-09T20:35:29.016 DEBUG:teuthology.orchestra.run:got remote process result: 7 2019-12-09T20:35:31.018 INFO:teuthology.orchestra.run.smithi067:> curl http://smithi067.front.sepia.ceph.com:80/ 2019-12-09T20:35:31.085 INFO:teuthology.orchestra.run.smithi067.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-09T20:35:31.091 INFO:teuthology.orchestra.run.smithi067.stderr: Dload Upload Total Spent Left Speed 2019-12-09T20:35:31.091 INFO:teuthology.orchestra.run.smithi067.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi067.front.sepia.ceph.com:80; Connection refused 2019-12-09T20:35:31.092 DEBUG:teuthology.orchestra.run:got remote process result: 7 2019-12-09T20:35:35.093 INFO:teuthology.orchestra.run.smithi067:> curl http://smithi067.front.sepia.ceph.com:80/ 2019-12-09T20:35:35.183 INFO:teuthology.orchestra.run.smithi067.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-09T20:35:35.183 INFO:teuthology.orchestra.run.smithi067.stderr: Dload Upload Total Spent Left Speed 2019-12-09T20:35:35.187 INFO:teuthology.orchestra.run.smithi067.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed connect to smithi067.front.sepia.ceph.com:80; Connection refused 2019-12-09T20:35:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 7 2019-12-09T20:35:43.189 INFO:teuthology.orchestra.run.smithi067:> curl http://smithi067.front.sepia.ceph.com:80/ 2019-12-09T20:35:43.243 INFO:teuthology.orchestra.run.smithi067.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2019-12-09T20:35:43.243 INFO:teuthology.orchestra.run.smithi067.stderr: Dload Upload Total Spent Left Speed 2019-12-09T20:35:44.243 INFO:teuthology.orchestra.run.smithi067.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 213 0 --:--:-- 0:00:01 --:--:-- 213 2019-12-09T20:35:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:anonymous 2019-12-09T20:35:44.244 INFO:teuthology.run_tasks:Running task workunit... 2019-12-09T20:35:44.278 INFO:tasks.workunit:Pulling workunits from ref 827e03c8a6b1e17d93dcaf79724437bac291c35b 2019-12-09T20:35:44.279 INFO:tasks.workunit:Making a separate scratch dir for every client... 2019-12-09T20:35:44.279 INFO:teuthology.orchestra.run.smithi067:> stat -- /home/ubuntu/cephtest/mnt.0 2019-12-09T20:35:44.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-09T20:35:44.335 INFO:teuthology.orchestra.run.smithi067.stderr:stat: cannot stat ‘/home/ubuntu/cephtest/mnt.0’: No such file or directory 2019-12-09T20:35:44.335 INFO:teuthology.orchestra.run.smithi067:> mkdir -- /home/ubuntu/cephtest/mnt.0 2019-12-09T20:35:44.381 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2019-12-09T20:35:44.381 INFO:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2019-12-09T20:35:44.424 INFO:tasks.workunit:timeout=3h 2019-12-09T20:35:44.424 INFO:tasks.workunit:cleanup=True 2019-12-09T20:35:44.425 INFO:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 827e03c8a6b1e17d93dcaf79724437bac291c35b 2019-12-09T20:35:44.734 INFO:tasks.workunit.client.0.smithi067.stdout:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2019-12-09T20:36:50.649 INFO:tasks.workunit.client.0.smithi067.stderr:Note: checking out '827e03c8a6b1e17d93dcaf79724437bac291c35b'. 2019-12-09T20:36:50.650 INFO:tasks.workunit.client.0.smithi067.stderr: 2019-12-09T20:36:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2019-12-09T20:36:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:changes and commit them, and you can discard any commits you make in this 2019-12-09T20:36:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:state without impacting any branches by performing another checkout. 2019-12-09T20:36:50.650 INFO:tasks.workunit.client.0.smithi067.stderr: 2019-12-09T20:36:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:If you want to create a new branch to retain commits you create, you may 2019-12-09T20:36:50.650 INFO:tasks.workunit.client.0.smithi067.stderr:do so (now or later) by using -b with the checkout command again. Example: 2019-12-09T20:36:50.651 INFO:tasks.workunit.client.0.smithi067.stderr: 2019-12-09T20:36:50.651 INFO:tasks.workunit.client.0.smithi067.stderr: git checkout -b new_branch_name 2019-12-09T20:36:50.651 INFO:tasks.workunit.client.0.smithi067.stderr: 2019-12-09T20:36:50.651 INFO:tasks.workunit.client.0.smithi067.stderr:HEAD is now at 827e03c... Merge branch 'rgw/public-buckets' of https://github.com/theanalyst/ceph into wip-abhi-testing-2019-12-04-1433 2019-12-09T20:36:50.653 INFO:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2019-12-09T20:36:50.746 INFO:tasks.workunit.client.0.smithi067.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2019-12-09T20:36:50.750 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-12-09T20:36:50.750 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2019-12-09T20:36:51.260 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2019-12-09T20:36:51.494 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2019-12-09T20:36:51.532 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-12-09T20:36:51.534 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-12-09T20:36:51.535 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2019-12-09T20:36:51.576 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-12-09T20:36:51.626 DEBUG:teuthology.orchestra.remote:smithi067:/home/ubuntu/cephtest/workunits.list.client.0 is 4KB 2019-12-09T20:36:51.676 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2019-12-09T20:36:51.676 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2019-12-09T20:36:51.676 INFO:teuthology.orchestra.run.smithi067:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=827e03c8a6b1e17d93dcaf79724437bac291c35b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2019-12-09T20:36:51.727 INFO:tasks.workunit.client.0.smithi067.stdout:Running main() from gmock_main.cc 2019-12-09T20:36:51.728 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] Running 11 tests from 1 test suite. 2019-12-09T20:36:51.728 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment set-up. 2019-12-09T20:36:51.728 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 11 tests from ClsLock 2019-12-09T20:36:51.728 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestMultiLocking 2019-12-09T20:36:53.996 INFO:tasks.workunit.client.0.smithi067.stdout:lock: mylock 2019-12-09T20:36:53.996 INFO:tasks.workunit.client.0.smithi067.stdout: lock_type: exclusive 2019-12-09T20:36:53.996 INFO:tasks.workunit.client.0.smithi067.stdout: tag: 2019-12-09T20:36:53.996 INFO:tasks.workunit.client.0.smithi067.stdout: lockers: 2019-12-09T20:36:53.996 INFO:tasks.workunit.client.0.smithi067.stdout: client.4343 expiration=2019-12-09T20:38:53.828763+0000 addr=v1:172.21.15.67:0/3003027034 cookie= 2019-12-09T20:36:54.211 INFO:tasks.workunit.client.0.smithi067.stdout:lock: mylock 2019-12-09T20:36:54.211 INFO:tasks.workunit.client.0.smithi067.stdout: lock_type: shared 2019-12-09T20:36:54.211 INFO:tasks.workunit.client.0.smithi067.stdout: tag: 2019-12-09T20:36:54.212 INFO:tasks.workunit.client.0.smithi067.stdout: lockers: 2019-12-09T20:36:54.212 INFO:tasks.workunit.client.0.smithi067.stdout: client.4343 expiration=2019-12-09T20:38:54.108504+0000 addr=v1:172.21.15.67:0/3003027034 cookie= 2019-12-09T20:36:54.212 INFO:tasks.workunit.client.0.smithi067.stdout: client.4575 expiration=0.000000 addr=v1:172.21.15.67:0/3172579519 cookie= 2019-12-09T20:36:54.259 INFO:tasks.workunit.client.0.smithi067.stdout:lock: mylock 2019-12-09T20:36:54.259 INFO:tasks.workunit.client.0.smithi067.stdout: lock_type: shared 2019-12-09T20:36:54.259 INFO:tasks.workunit.client.0.smithi067.stdout: tag: 2019-12-09T20:36:54.259 INFO:tasks.workunit.client.0.smithi067.stdout: lockers: 2019-12-09T20:36:54.260 INFO:tasks.workunit.client.0.smithi067.stdout: client.4575 expiration=0.000000 addr=v1:172.21.15.67:0/3172579519 cookie= 2019-12-09T20:36:55.409 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestMultiLocking (3681 ms) 2019-12-09T20:36:55.409 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestMeta 2019-12-09T20:36:58.595 INFO:tasks.workunit.client.0.smithi067.stdout:lock: mylock 2019-12-09T20:36:58.595 INFO:tasks.workunit.client.0.smithi067.stdout: lock_type: shared 2019-12-09T20:36:58.595 INFO:tasks.workunit.client.0.smithi067.stdout: tag: 2019-12-09T20:36:58.596 INFO:tasks.workunit.client.0.smithi067.stdout: lockers: 2019-12-09T20:36:58.596 INFO:tasks.workunit.client.0.smithi067.stdout: client.4480 expiration=0.000000 addr=v1:172.21.15.67:0/472793706 cookie= 2019-12-09T20:36:58.723 INFO:tasks.workunit.client.0.smithi067.stdout:lock: mylock 2019-12-09T20:36:58.724 INFO:tasks.workunit.client.0.smithi067.stdout: lock_type: exclusive 2019-12-09T20:36:58.724 INFO:tasks.workunit.client.0.smithi067.stdout: tag: new_tag 2019-12-09T20:36:58.724 INFO:tasks.workunit.client.0.smithi067.stdout: lockers: 2019-12-09T20:36:58.725 INFO:tasks.workunit.client.0.smithi067.stdout: client.4352 expiration=0.000000 addr=v1:172.21.15.67:0/1092403350 cookie= 2019-12-09T20:36:59.898 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestMeta (4486 ms) 2019-12-09T20:36:59.898 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestCookie 2019-12-09T20:37:02.939 INFO:tasks.workunit.client.0.smithi067.stdout:lock: mylock 2019-12-09T20:37:02.939 INFO:tasks.workunit.client.0.smithi067.stdout: lock_type: exclusive 2019-12-09T20:37:02.939 INFO:tasks.workunit.client.0.smithi067.stdout: tag: 2019-12-09T20:37:02.939 INFO:tasks.workunit.client.0.smithi067.stdout: lockers: 2019-12-09T20:37:03.031 INFO:tasks.workunit.client.0.smithi067.stdout:lock: mylock 2019-12-09T20:37:03.031 INFO:tasks.workunit.client.0.smithi067.stdout: lock_type: shared 2019-12-09T20:37:03.031 INFO:tasks.workunit.client.0.smithi067.stdout: tag: 2019-12-09T20:37:03.031 INFO:tasks.workunit.client.0.smithi067.stdout: lockers: 2019-12-09T20:37:03.031 INFO:tasks.workunit.client.0.smithi067.stdout: client.4489 expiration=0.000000 addr=v1:172.21.15.67:0/2986258326 cookie= 2019-12-09T20:37:03.031 INFO:tasks.workunit.client.0.smithi067.stdout: client.4489 expiration=0.000000 addr=v1:172.21.15.67:0/2986258326 cookie=new cookie 2019-12-09T20:37:03.312 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestCookie (3417 ms) 2019-12-09T20:37:03.312 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestMultipleLocks 2019-12-09T20:37:06.854 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestMultipleLocks (3542 ms) 2019-12-09T20:37:06.854 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestLockDuration 2019-12-09T20:37:15.296 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestLockDuration (8440 ms) 2019-12-09T20:37:15.296 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestAssertLocked 2019-12-09T20:37:18.751 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestAssertLocked (3451 ms) 2019-12-09T20:37:18.752 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestSetCookie 2019-12-09T20:37:23.335 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestSetCookie (4589 ms) 2019-12-09T20:37:23.335 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestRenew 2019-12-09T20:37:45.394 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestRenew (22057 ms) 2019-12-09T20:37:45.395 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2019-12-09T20:37:52.625 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (7224 ms) 2019-12-09T20:37:52.626 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2019-12-09T20:38:01.078 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (8460 ms) 2019-12-09T20:38:01.078 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2019-12-09T20:38:09.799 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8719 ms) 2019-12-09T20:38:09.800 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 11 tests from ClsLock (78066 ms total) 2019-12-09T20:38:09.800 INFO:tasks.workunit.client.0.smithi067.stdout: 2019-12-09T20:38:09.800 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment tear-down 2019-12-09T20:38:09.800 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] 11 tests from 1 test suite ran. (78066 ms total) 2019-12-09T20:38:09.800 INFO:tasks.workunit.client.0.smithi067.stdout:[ PASSED ] 11 tests. 2019-12-09T20:38:09.801 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-09T20:38:09.802 INFO:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-09T20:38:09.909 INFO:tasks.workunit:Running workunits matching cls/test_cls_log.sh on client.0... 2019-12-09T20:38:09.909 INFO:tasks.workunit:Running workunit cls/test_cls_log.sh... 2019-12-09T20:38:09.909 INFO:teuthology.orchestra.run.smithi067:workunit test cls/test_cls_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=827e03c8a6b1e17d93dcaf79724437bac291c35b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_log.sh 2019-12-09T20:38:10.001 INFO:tasks.workunit.client.0.smithi067.stdout:Running main() from gmock_main.cc 2019-12-09T20:38:10.001 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] Running 4 tests from 1 test suite. 2019-12-09T20:38:10.001 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment set-up. 2019-12-09T20:38:10.001 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 4 tests from cls_log 2019-12-09T20:38:10.001 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_log.test_log_add_same_time 2019-12-09T20:38:13.439 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_log.test_log_add_same_time (3437 ms) 2019-12-09T20:38:13.439 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_log.test_log_add_different_time 2019-12-09T20:38:16.959 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_log.test_log_add_different_time (3521 ms) 2019-12-09T20:38:16.959 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_log.trim_by_time 2019-12-09T20:38:22.432 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_log.trim_by_time (5472 ms) 2019-12-09T20:38:22.432 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_log.trim_by_marker 2019-12-09T20:38:26.093 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_log.trim_by_marker (3662 ms) 2019-12-09T20:38:26.093 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 4 tests from cls_log (16092 ms total) 2019-12-09T20:38:26.094 INFO:tasks.workunit.client.0.smithi067.stdout: 2019-12-09T20:38:26.094 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment tear-down 2019-12-09T20:38:26.094 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] 4 tests from 1 test suite ran. (16092 ms total) 2019-12-09T20:38:26.094 INFO:tasks.workunit.client.0.smithi067.stdout:[ PASSED ] 4 tests. 2019-12-09T20:38:26.095 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-09T20:38:26.096 INFO:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-09T20:38:26.163 INFO:tasks.workunit:Running workunits matching cls/test_cls_refcount.sh on client.0... 2019-12-09T20:38:26.163 INFO:tasks.workunit:Running workunit cls/test_cls_refcount.sh... 2019-12-09T20:38:26.163 INFO:teuthology.orchestra.run.smithi067:workunit test cls/test_cls_refcount.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=827e03c8a6b1e17d93dcaf79724437bac291c35b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh 2019-12-09T20:38:26.215 INFO:tasks.workunit.client.0.smithi067.stdout:Running main() from gmock_main.cc 2019-12-09T20:38:26.216 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] Running 5 tests from 1 test suite. 2019-12-09T20:38:26.216 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment set-up. 2019-12-09T20:38:26.216 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 5 tests from cls_rgw 2019-12-09T20:38:26.217 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.test_implicit 2019-12-09T20:38:29.361 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.test_implicit (3145 ms) 2019-12-09T20:38:29.361 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.test_implicit_idempotent 2019-12-09T20:38:32.901 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.test_implicit_idempotent (3533 ms) 2019-12-09T20:38:32.901 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.test_put_snap 2019-12-09T20:38:42.267 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-09T20:38:42.177+0000 37f93700 -1 osd.2 pg_epoch: 108 pg[25.6( v 107'4 (0'0,107'4] local-lis/les=104/105 n=2 ec=104/104 lis/c=104/104 les/c/f=105/105/0 sis=104) [2,1] r=0 lpr=104 crt=106'3 lcod 106'3 mlcod 106'3 active+clean+snaptrim trimq=[1~1]] removing snap head 2019-12-09T20:38:43.279 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.test_put_snap (10385 ms) 2019-12-09T20:38:43.280 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.test_explicit 2019-12-09T20:38:47.051 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.test_explicit (3772 ms) 2019-12-09T20:38:47.052 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.set 2019-12-09T20:38:50.490 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.set (3439 ms) 2019-12-09T20:38:50.491 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 5 tests from cls_rgw (24274 ms total) 2019-12-09T20:38:50.491 INFO:tasks.workunit.client.0.smithi067.stdout: 2019-12-09T20:38:50.491 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment tear-down 2019-12-09T20:38:50.491 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] 5 tests from 1 test suite ran. (24274 ms total) 2019-12-09T20:38:50.491 INFO:tasks.workunit.client.0.smithi067.stdout:[ PASSED ] 5 tests. 2019-12-09T20:38:50.492 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-09T20:38:50.492 INFO:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-09T20:38:50.611 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw.sh on client.0... 2019-12-09T20:38:50.611 INFO:tasks.workunit:Running workunit cls/test_cls_rgw.sh... 2019-12-09T20:38:50.612 INFO:teuthology.orchestra.run.smithi067:workunit test cls/test_cls_rgw.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=827e03c8a6b1e17d93dcaf79724437bac291c35b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh 2019-12-09T20:38:50.675 INFO:tasks.workunit.client.0.smithi067.stdout:Running main() from gmock_main.cc 2019-12-09T20:38:50.675 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] Running 13 tests from 1 test suite. 2019-12-09T20:38:50.675 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment set-up. 2019-12-09T20:38:50.675 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 13 tests from cls_rgw 2019-12-09T20:38:52.737 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.index_basic 2019-12-09T20:38:54.438 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.index_basic (1701 ms) 2019-12-09T20:38:54.438 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.index_multiple_obj_writers 2019-12-09T20:38:56.281 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.index_multiple_obj_writers (1843 ms) 2019-12-09T20:38:56.281 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.index_remove_object 2019-12-09T20:38:58.363 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.index_remove_object (2082 ms) 2019-12-09T20:38:58.364 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.index_suggest 2019-12-09T20:39:25.244 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.index_suggest (26879 ms) 2019-12-09T20:39:25.244 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.index_list 2019-12-09T20:39:26.824 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.index_list (1580 ms) 2019-12-09T20:39:26.824 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.bi_list 2019-12-09T20:39:29.838 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.bi_list (3014 ms) 2019-12-09T20:39:29.839 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.gc_set 2019-12-09T20:39:30.388 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.gc_set (549 ms) 2019-12-09T20:39:30.388 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.gc_list 2019-12-09T20:39:31.139 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.gc_list (751 ms) 2019-12-09T20:39:31.140 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.gc_defer 2019-12-09T20:39:39.874 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.gc_defer (8735 ms) 2019-12-09T20:39:39.875 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.usage_basic 2019-12-09T20:39:47.645 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.usage_basic (7770 ms) 2019-12-09T20:39:47.645 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.usage_clear_no_obj 2019-12-09T20:39:47.716 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.usage_clear_no_obj (67 ms) 2019-12-09T20:39:47.716 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.usage_clear 2019-12-09T20:39:57.370 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.usage_clear (9657 ms) 2019-12-09T20:39:57.370 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] cls_rgw.bi_log_trim 2019-12-09T20:39:59.458 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] cls_rgw.bi_log_trim (2088 ms) 2019-12-09T20:39:59.838 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 13 tests from cls_rgw (66717 ms total) 2019-12-09T20:39:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: 2019-12-09T20:39:59.838 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment tear-down 2019-12-09T20:39:59.838 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] 13 tests from 1 test suite ran. (69159 ms total) 2019-12-09T20:39:59.838 INFO:tasks.workunit.client.0.smithi067.stdout:[ PASSED ] 13 tests. 2019-12-09T20:39:59.840 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-09T20:39:59.840 INFO:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-09T20:39:59.959 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_gc_log.sh on client.0... 2019-12-09T20:39:59.959 INFO:tasks.workunit:Running workunit rgw/test_rgw_gc_log.sh... 2019-12-09T20:39:59.960 INFO:teuthology.orchestra.run.smithi067:workunit test rgw/test_rgw_gc_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=827e03c8a6b1e17d93dcaf79724437bac291c35b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_gc_log.sh 2019-12-09T20:40:00.033 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] Running 6 tests from 1 test suite. 2019-12-09T20:40:00.033 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment set-up. 2019-12-09T20:40:02.207 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 6 tests from rgw_gc_log 2019-12-09T20:40:02.207 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] rgw_gc_log.init_existing_queue 2019-12-09T20:40:02.589 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] rgw_gc_log.init_existing_queue (382 ms) 2019-12-09T20:40:02.589 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] rgw_gc_log.init_existing_omap 2019-12-09T20:40:02.937 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] rgw_gc_log.init_existing_omap (348 ms) 2019-12-09T20:40:02.937 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] rgw_gc_log.enqueue1_after_init 2019-12-09T20:40:03.060 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] rgw_gc_log.enqueue1_after_init (123 ms) 2019-12-09T20:40:03.061 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] rgw_gc_log.enqueue2_before_init 2019-12-09T20:40:03.136 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] rgw_gc_log.enqueue2_before_init (76 ms) 2019-12-09T20:40:03.137 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] rgw_gc_log.defer1_after_init 2019-12-09T20:40:03.269 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] rgw_gc_log.defer1_after_init (133 ms) 2019-12-09T20:40:03.269 INFO:tasks.workunit.client.0.smithi067.stdout:[ RUN ] rgw_gc_log.defer2_before_init 2019-12-09T20:40:03.314 INFO:tasks.workunit.client.0.smithi067.stdout:[ OK ] rgw_gc_log.defer2_before_init (45 ms) 2019-12-09T20:40:03.314 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] 6 tests from rgw_gc_log (1107 ms total) 2019-12-09T20:40:03.315 INFO:tasks.workunit.client.0.smithi067.stdout: 2019-12-09T20:40:03.315 INFO:tasks.workunit.client.0.smithi067.stdout:[----------] Global test environment tear-down 2019-12-09T20:40:04.365 INFO:tasks.workunit.client.0.smithi067.stdout:[==========] 6 tests from 1 test suite ran. (4332 ms total) 2019-12-09T20:40:04.365 INFO:tasks.workunit.client.0.smithi067.stdout:[ PASSED ] 6 tests. 2019-12-09T20:40:04.368 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-09T20:40:04.368 INFO:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-09T20:40:04.488 INFO:tasks.workunit:Stopping ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'rgw/test_rgw_gc_log.sh'] on client.0... 2019-12-09T20:40:04.488 INFO:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2019-12-09T20:40:04.710 DEBUG:teuthology.parallel:result is None 2019-12-09T20:40:04.710 INFO:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2019-12-09T20:40:04.770 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2019-12-09T20:40:04.770 INFO:teuthology.orchestra.run.smithi067:> rmdir -- /home/ubuntu/cephtest/mnt.0 2019-12-09T20:40:04.815 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2019-12-09T20:40:04.816 INFO:teuthology.run_tasks:Running task ragweed... 2019-12-09T20:40:04.840 DEBUG:tasks.ragweed:ragweed config is {'client.0': {'rgw_server': 'client.0', 'stages': 'prepare,check', 'default-branch': 'ceph-master'}} 2019-12-09T20:40:04.840 INFO:tasks.ragweed:Downloading ragweed... 2019-12-09T20:40:04.840 INFO:tasks.ragweed:Using branch 'wip-abhi-testing-2019-12-04-1433' for ragweed 2019-12-09T20:40:04.841 INFO:teuthology.orchestra.run.smithi067:> git clone -b wip-abhi-testing-2019-12-04-1433 git://git.ceph.com/git/ragweed.git /home/ubuntu/cephtest/ragweed 2019-12-09T20:40:04.895 INFO:teuthology.orchestra.run.smithi067.stderr:fatal: Remote branch wip-abhi-testing-2019-12-04-1433 not found in upstream origin 2019-12-09T20:40:04.896 INFO:teuthology.orchestra.run.smithi067.stdout:Cloning into '/home/ubuntu/cephtest/ragweed'... 2019-12-09T20:40:04.896 DEBUG:teuthology.orchestra.run:got remote process result: 128 2019-12-09T20:40:04.896 INFO:teuthology.orchestra.run.smithi067:> git clone -b ceph-master git://git.ceph.com/git/ragweed.git /home/ubuntu/cephtest/ragweed 2019-12-09T20:40:04.993 INFO:teuthology.orchestra.run.smithi067.stdout:Cloning into '/home/ubuntu/cephtest/ragweed'... 2019-12-09T20:40:05.090 INFO:tasks.ragweed:Creating rgw users... 2019-12-09T20:40:05.091 DEBUG:tasks.ragweed:Creating user sysuser.client.0 on client.0 2019-12-09T20:40:05.091 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid sysuser.client.0 --display-name 'Mr. sysuser.client.0' --access-key QHAWDUANSLCTQKJUAZPK --secret v0Rq3NXHvyB93ssloyc9bBmawMlrrCSEUe1NJJi484KV/hnfgHZwmQ== --email sysuser.client.0+test@test.test --system true 2019-12-09T20:40:05.236 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.202+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.264 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.231+0000 7fe4990a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:05.265 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.231+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.240+0000 7fe4990a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:05.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.240+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.245+0000 7fe4990a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:05.279 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.245+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.292 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.259+0000 7fe4990a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:05.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.573+0000 7fe4990a7ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T20:40:05.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.573+0000 7fe4990a7ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T20:40:05.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.573+0000 7fe4990a7ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T20:40:05.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.573+0000 7fe4990a7ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T20:40:05.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.573+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.580+0000 7fe4990a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:05.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.580+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.589+0000 7fe4990a7ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:40:05.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.589+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.597+0000 7fe4990a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:05.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.597+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.611+0000 7fe4990a7ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:40:05.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.611+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.621+0000 7fe4990a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:05.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.621+0000 7fe4990a7ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T20:40:05.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.621+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.626+0000 7fe4990a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:05.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.626+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.634+0000 7fe4990a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:05.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.634+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.645+0000 7fe4990a7ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:40:05.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.645+0000 7fe4990a7ec0 20 zonegroup default 2019-12-09T20:40:05.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.645+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.682 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.649+0000 7fe4990a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:05.682 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.649+0000 7fe4990a7ec0 10 Cannot find current period zone using local zone 2019-12-09T20:40:05.683 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.649+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.653+0000 7fe4990a7ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:05.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.653+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.696 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.661+0000 7fe4990a7ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:05.696 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.661+0000 7fe4990a7ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.670+0000 7fe4990a7ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:40:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.670+0000 7fe4990a7ec0 20 zone default found 2019-12-09T20:40:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.670+0000 7fe4990a7ec0 20 started sync module instance, tier type = 2019-12-09T20:40:05.704 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:05.670+0000 7fe4990a7ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T20:40:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.002+0000 7fe4990a7ec0 20 add_watcher() i=0 2019-12-09T20:40:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.002+0000 7fe4990a7ec0 20 add_watcher() i=1 2019-12-09T20:40:06.037 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.002+0000 7fe4990a7ec0 20 add_watcher() i=2 2019-12-09T20:40:06.037 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.002+0000 7fe4990a7ec0 20 add_watcher() i=3 2019-12-09T20:40:06.037 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.002+0000 7fe4990a7ec0 20 add_watcher() i=4 2019-12-09T20:40:06.037 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.002+0000 7fe4990a7ec0 20 add_watcher() i=5 2019-12-09T20:40:06.067 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.033+0000 7fe4990a7ec0 20 add_watcher() i=6 2019-12-09T20:40:06.076 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.042+0000 7fe4990a7ec0 20 add_watcher() i=7 2019-12-09T20:40:06.076 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.042+0000 7fe4990a7ec0 2 all 8 watchers are set, enabling cache 2019-12-09T20:40:06.076 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.042+0000 7fe456fdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:40:06.089 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.055+0000 7fe4862d1700 0 -- 172.21.15.67:0/2648629692 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x556949be8680 msgr2=0x556949beda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:40:06.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:06.093+0000 7fe4862d1700 0 -- 172.21.15.67:0/3767006392 >> [v2:172.21.15.67:6800/9311,v1:172.21.15.67:6801/9311] conn(0x7fe3c80224b0 msgr2=0x7fe3c8024980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-09T20:40:07.737 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.702+0000 7fe3c57fa700 20 reqs_thread_entry: start 2019-12-09T20:40:07.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.702+0000 7fe3beffd700 20 reqs_thread_entry: start 2019-12-09T20:40:07.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.702+0000 7fe4990a7ec0 20 init_complete bucket index max shards: 0 2019-12-09T20:40:07.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.702+0000 7fe3bd7fa700 20 reqs_thread_entry: start 2019-12-09T20:40:07.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.702+0000 7fe4990a7ec0 20 get_system_obj_state: rctx=0x7ffcd0c7a088 obj=default.rgw.meta:users.uid:sysuser.client.0 state=0x556949c6e3f0 s->prefetch_data=0 2019-12-09T20:40:07.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.702+0000 7fe4990a7ec0 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2019-12-09T20:40:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.716+0000 7fe4990a7ec0 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x0 2019-12-09T20:40:07.750 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.716+0000 7fe4990a7ec0 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-09T20:40:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.716+0000 7fe4990a7ec0 20 get_system_obj_state: rctx=0x7ffcd0c7a098 obj=default.rgw.meta:users.email:sysuser.client.0+test@test.test state=0x556949c70180 s->prefetch_data=0 2019-12-09T20:40:07.751 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.716+0000 7fe4990a7ec0 10 cache get: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test : miss 2019-12-09T20:40:07.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.737+0000 7fe4990a7ec0 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x0 2019-12-09T20:40:07.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.737+0000 7fe4990a7ec0 10 adding default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2019-12-09T20:40:07.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.737+0000 7fe4990a7ec0 20 get_system_obj_state: rctx=0x7ffcd0c7a098 obj=default.rgw.meta:users.keys:QHAWDUANSLCTQKJUAZPK state=0x556949c70180 s->prefetch_data=0 2019-12-09T20:40:07.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.737+0000 7fe4990a7ec0 10 cache get: name=default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK : miss 2019-12-09T20:40:07.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.749+0000 7fe4990a7ec0 10 cache put: name=default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK info.flags=0x0 2019-12-09T20:40:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.749+0000 7fe4990a7ec0 10 adding default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK to cache LRU end 2019-12-09T20:40:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.749+0000 7fe4990a7ec0 20 get_system_obj_state: rctx=0x7ffcd0c79ca8 obj=default.rgw.meta:users.keys:QHAWDUANSLCTQKJUAZPK state=0x556949c70180 s->prefetch_data=0 2019-12-09T20:40:07.784 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.749+0000 7fe4990a7ec0 10 cache get: name=default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK : type miss (requested=0x6, cached=0x0) 2019-12-09T20:40:07.793 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.758+0000 7fe4990a7ec0 10 cache put: name=default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK info.flags=0x0 2019-12-09T20:40:07.793 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.758+0000 7fe4990a7ec0 10 moving default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK to cache LRU end 2019-12-09T20:40:07.793 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.758+0000 7fe4990a7ec0 20 get_system_obj_state: rctx=0x7ffcd0c79e28 obj=default.rgw.meta:users.keys:QHAWDUANSLCTQKJUAZPK state=0x556949c70180 s->prefetch_data=0 2019-12-09T20:40:07.793 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.758+0000 7fe4990a7ec0 10 cache get: name=default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK : type miss (requested=0x6, cached=0x0) 2019-12-09T20:40:07.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.767+0000 7fe4990a7ec0 10 cache put: name=default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK info.flags=0x0 2019-12-09T20:40:07.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.767+0000 7fe4990a7ec0 10 moving default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK to cache LRU end 2019-12-09T20:40:07.869 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.835+0000 7fe4990a7ec0 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2019-12-09T20:40:07.869 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.835+0000 7fe4990a7ec0 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-09T20:40:07.869 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.835+0000 7fe4990a7ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=637 2019-12-09T20:40:07.923 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.889+0000 7fe4677fe700 10 RGWWatcher::handle_notify() notify_id 549755813888 cookie 93910697056208 notifier 4585 bl.length()=637 2019-12-09T20:40:07.924 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.889+0000 7fe4677fe700 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2019-12-09T20:40:07.924 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:07.889+0000 7fe4677fe700 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-09T20:40:08.156 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.123+0000 7fe4990a7ec0 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2019-12-09T20:40:08.156 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.123+0000 7fe4990a7ec0 10 moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2019-12-09T20:40:08.157 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.123+0000 7fe4990a7ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=202 2019-12-09T20:40:08.190 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.157+0000 7fe4677fe700 10 RGWWatcher::handle_notify() notify_id 549755813888 cookie 93910697078800 notifier 4585 bl.length()=202 2019-12-09T20:40:08.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.157+0000 7fe4677fe700 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2019-12-09T20:40:08.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.157+0000 7fe4677fe700 10 moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2019-12-09T20:40:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.678+0000 7fe4990a7ec0 10 cache put: name=default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK info.flags=0x7 2019-12-09T20:40:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.678+0000 7fe4990a7ec0 10 moving default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK to cache LRU end 2019-12-09T20:40:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.678+0000 7fe4990a7ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=190 2019-12-09T20:40:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.770+0000 7fe4677fe700 10 RGWWatcher::handle_notify() notify_id 549755813888 cookie 93910697055280 notifier 4585 bl.length()=190 2019-12-09T20:40:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.771+0000 7fe4677fe700 10 cache put: name=default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK info.flags=0x7 2019-12-09T20:40:08.804 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:08.771+0000 7fe4677fe700 10 moving default.rgw.meta+users.keys+QHAWDUANSLCTQKJUAZPK to cache LRU end 2019-12-09T20:40:09.167 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2019-12-09T20:40:09.167 INFO:teuthology.orchestra.run.smithi067.stdout: "user_id": "sysuser.client.0", 2019-12-09T20:40:09.167 INFO:teuthology.orchestra.run.smithi067.stdout: "display_name": "Mr. sysuser.client.0", 2019-12-09T20:40:09.167 INFO:teuthology.orchestra.run.smithi067.stdout: "email": "sysuser.client.0+test@test.test", 2019-12-09T20:40:09.168 INFO:teuthology.orchestra.run.smithi067.stdout: "suspended": 0, 2019-12-09T20:40:09.168 INFO:teuthology.orchestra.run.smithi067.stdout: "max_buckets": 1000, 2019-12-09T20:40:09.168 INFO:teuthology.orchestra.run.smithi067.stdout: "subusers": [], 2019-12-09T20:40:09.168 INFO:teuthology.orchestra.run.smithi067.stdout: "keys": [ 2019-12-09T20:40:09.168 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-09T20:40:09.168 INFO:teuthology.orchestra.run.smithi067.stdout: "user": "sysuser.client.0", 2019-12-09T20:40:09.169 INFO:teuthology.orchestra.run.smithi067.stdout: "access_key": "QHAWDUANSLCTQKJUAZPK", 2019-12-09T20:40:09.169 INFO:teuthology.orchestra.run.smithi067.stdout: "secret_key": "v0Rq3NXHvyB93ssloyc9bBmawMlrrCSEUe1NJJi484KV/hnfgHZwmQ==" 2019-12-09T20:40:09.169 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-09T20:40:09.169 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-09T20:40:09.169 INFO:teuthology.orchestra.run.smithi067.stdout: "swift_keys": [], 2019-12-09T20:40:09.169 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": [], 2019-12-09T20:40:09.169 INFO:teuthology.orchestra.run.smithi067.stdout: "op_mask": "read, write, delete", 2019-12-09T20:40:09.170 INFO:teuthology.orchestra.run.smithi067.stdout: "system": "true", 2019-12-09T20:40:09.170 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "", 2019-12-09T20:40:09.170 INFO:teuthology.orchestra.run.smithi067.stdout: "default_storage_class": "", 2019-12-09T20:40:09.170 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_tags": [], 2019-12-09T20:40:09.170 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_quota": { 2019-12-09T20:40:09.170 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T20:40:09.170 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T20:40:09.170 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T20:40:09.171 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T20:40:09.171 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T20:40:09.171 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T20:40:09.171 INFO:teuthology.orchestra.run.smithi067.stdout: "user_quota": { 2019-12-09T20:40:09.171 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T20:40:09.171 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T20:40:09.171 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T20:40:09.172 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T20:40:09.172 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T20:40:09.172 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T20:40:09.173 INFO:teuthology.orchestra.run.smithi067.stdout: "temp_url_keys": [], 2019-12-09T20:40:09.174 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "rgw", 2019-12-09T20:40:09.174 INFO:teuthology.orchestra.run.smithi067.stdout: "mfa_ids": [] 2019-12-09T20:40:09.174 INFO:teuthology.orchestra.run.smithi067.stdout:} 2019-12-09T20:40:09.174 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:09.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.194+0000 7fe4990a7ec0 20 remove_watcher() i=0 2019-12-09T20:40:09.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.194+0000 7fe4990a7ec0 2 removed watcher, disabling cache 2019-12-09T20:40:09.298 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.263+0000 7fe4990a7ec0 20 remove_watcher() i=1 2019-12-09T20:40:09.371 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.338+0000 7fe4990a7ec0 20 remove_watcher() i=2 2019-12-09T20:40:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.413+0000 7fe4990a7ec0 20 remove_watcher() i=3 2019-12-09T20:40:09.503 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.470+0000 7fe4990a7ec0 20 remove_watcher() i=4 2019-12-09T20:40:09.571 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.538+0000 7fe4990a7ec0 20 remove_watcher() i=5 2019-12-09T20:40:09.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.610+0000 7fe4990a7ec0 20 remove_watcher() i=6 2019-12-09T20:40:09.720 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.687+0000 7fe4990a7ec0 20 remove_watcher() i=7 2019-12-09T20:40:09.732 DEBUG:tasks.ragweed:Creating user ragweed.client.0 on client.0 2019-12-09T20:40:09.733 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid ragweed.client.0 --display-name 'Mr. ragweed.client.0' --access-key JBDGQWYZIKQZJAIXHECI --secret 3Y9xy32//HKFJ2+tAzjg8NtuZ72tIZzokxqu277vrxn8fpItcZfazQ== --email ragweed.client.0+test@test.test --system false 2019-12-09T20:40:09.892 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.859+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:09.921 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.886+0000 7fbc34440ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:09.921 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.886+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:09.929 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.896+0000 7fbc34440ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:09.930 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.896+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:09.935 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.902+0000 7fbc34440ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:09.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.902+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:09.911+0000 7fbc34440ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:10.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.096+0000 7fbc34440ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T20:40:10.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.096+0000 7fbc34440ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T20:40:10.130 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.096+0000 7fbc34440ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T20:40:10.130 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.096+0000 7fbc34440ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T20:40:10.130 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.096+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.104+0000 7fbc34440ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:10.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.104+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.112+0000 7fbc34440ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:40:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.112+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.153 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.120+0000 7fbc34440ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:10.154 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.120+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.165 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.132+0000 7fbc34440ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:40:10.165 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.132+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.138+0000 7fbc34440ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.138+0000 7fbc34440ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T20:40:10.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.138+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.178 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.145+0000 7fbc34440ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:10.178 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.145+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.152+0000 7fbc34440ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:10.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.152+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.161+0000 7fbc34440ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:40:10.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.161+0000 7fbc34440ec0 20 zonegroup default 2019-12-09T20:40:10.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.161+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.166+0000 7fbc34440ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.166+0000 7fbc34440ec0 10 Cannot find current period zone using local zone 2019-12-09T20:40:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.166+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.172+0000 7fbc34440ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:40:10.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.172+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.217 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.181+0000 7fbc34440ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:40:10.217 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.182+0000 7fbc34440ec0 20 rados->read ofs=0 len=0 2019-12-09T20:40:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.196+0000 7fbc34440ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:40:10.229 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.196+0000 7fbc34440ec0 20 zone default found 2019-12-09T20:40:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.196+0000 7fbc34440ec0 20 started sync module instance, tier type = 2019-12-09T20:40:10.230 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.196+0000 7fbc34440ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T20:40:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.572+0000 7fbc34440ec0 20 add_watcher() i=0 2019-12-09T20:40:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.572+0000 7fbc34440ec0 20 add_watcher() i=1 2019-12-09T20:40:10.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.572+0000 7fbc34440ec0 20 add_watcher() i=2 2019-12-09T20:40:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.572+0000 7fbc34440ec0 20 add_watcher() i=3 2019-12-09T20:40:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.572+0000 7fbc34440ec0 20 add_watcher() i=4 2019-12-09T20:40:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.572+0000 7fbc34440ec0 20 add_watcher() i=5 2019-12-09T20:40:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.572+0000 7fbc34440ec0 20 add_watcher() i=6 2019-12-09T20:40:10.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.606+0000 7fbc34440ec0 20 add_watcher() i=7 2019-12-09T20:40:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.606+0000 7fbc34440ec0 2 all 8 watchers are set, enabling cache 2019-12-09T20:40:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.606+0000 7fbbf1fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:40:10.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.608+0000 7fbc1bfff700 0 -- 172.21.15.67:0/3373553967 >> v1:172.21.15.67:6789/0 conn(0x55dac1181d00 legacy=0x55dac11841d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:40:10.680 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:10.647+0000 7fbc2166a700 0 -- 172.21.15.67:0/4006579460 >> [v2:172.21.15.67:6800/9311,v1:172.21.15.67:6801/9311] conn(0x7fbb64022480 msgr2=0x7fbb64024990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:40:12.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.058+0000 7fbb63fff700 20 reqs_thread_entry: start 2019-12-09T20:40:12.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.058+0000 7fbb61ffb700 20 reqs_thread_entry: start 2019-12-09T20:40:12.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.058+0000 7fbc34440ec0 20 init_complete bucket index max shards: 0 2019-12-09T20:40:12.093 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.059+0000 7fbb4ffff700 20 reqs_thread_entry: start 2019-12-09T20:40:12.093 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.059+0000 7fbc34440ec0 20 get_system_obj_state: rctx=0x7ffc4fb83928 obj=default.rgw.meta:users.uid:ragweed.client.0 state=0x55dac1200c80 s->prefetch_data=0 2019-12-09T20:40:12.093 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.059+0000 7fbc34440ec0 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2019-12-09T20:40:12.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.068+0000 7fbc34440ec0 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x0 2019-12-09T20:40:12.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.068+0000 7fbc34440ec0 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-09T20:40:12.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.068+0000 7fbc34440ec0 20 get_system_obj_state: rctx=0x7ffc4fb83938 obj=default.rgw.meta:users.email:ragweed.client.0+test@test.test state=0x55dac1202a10 s->prefetch_data=0 2019-12-09T20:40:12.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.068+0000 7fbc34440ec0 10 cache get: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test : miss 2019-12-09T20:40:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.075+0000 7fbc34440ec0 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x0 2019-12-09T20:40:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.075+0000 7fbc34440ec0 10 adding default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2019-12-09T20:40:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.075+0000 7fbc34440ec0 20 get_system_obj_state: rctx=0x7ffc4fb83938 obj=default.rgw.meta:users.keys:JBDGQWYZIKQZJAIXHECI state=0x55dac1202a10 s->prefetch_data=0 2019-12-09T20:40:12.109 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.075+0000 7fbc34440ec0 10 cache get: name=default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI : miss 2019-12-09T20:40:12.125 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.081+0000 7fbc34440ec0 10 cache put: name=default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI info.flags=0x0 2019-12-09T20:40:12.125 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.081+0000 7fbc34440ec0 10 adding default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI to cache LRU end 2019-12-09T20:40:12.125 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.081+0000 7fbc34440ec0 20 get_system_obj_state: rctx=0x7ffc4fb83548 obj=default.rgw.meta:users.keys:JBDGQWYZIKQZJAIXHECI state=0x55dac1202a10 s->prefetch_data=0 2019-12-09T20:40:12.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.081+0000 7fbc34440ec0 10 cache get: name=default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI : type miss (requested=0x6, cached=0x0) 2019-12-09T20:40:12.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.086+0000 7fbc34440ec0 10 cache put: name=default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI info.flags=0x0 2019-12-09T20:40:12.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.086+0000 7fbc34440ec0 10 moving default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI to cache LRU end 2019-12-09T20:40:12.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.087+0000 7fbc34440ec0 20 get_system_obj_state: rctx=0x7ffc4fb836c8 obj=default.rgw.meta:users.keys:JBDGQWYZIKQZJAIXHECI state=0x55dac1202a10 s->prefetch_data=0 2019-12-09T20:40:12.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.087+0000 7fbc34440ec0 10 cache get: name=default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI : type miss (requested=0x6, cached=0x0) 2019-12-09T20:40:12.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.090+0000 7fbc34440ec0 10 cache put: name=default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI info.flags=0x0 2019-12-09T20:40:12.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.090+0000 7fbc34440ec0 10 moving default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI to cache LRU end 2019-12-09T20:40:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.134+0000 7fbc34440ec0 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2019-12-09T20:40:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.134+0000 7fbc34440ec0 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-09T20:40:12.169 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.134+0000 7fbc34440ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=637 2019-12-09T20:40:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.154+0000 7fbc027fc700 10 RGWWatcher::handle_notify() notify_id 549755813889 cookie 94398030719952 notifier 4436 bl.length()=637 2019-12-09T20:40:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.154+0000 7fbc027fc700 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2019-12-09T20:40:12.187 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.154+0000 7fbc027fc700 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-09T20:40:12.514 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.481+0000 7fbc34440ec0 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2019-12-09T20:40:12.515 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.481+0000 7fbc34440ec0 10 moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2019-12-09T20:40:12.515 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.481+0000 7fbc34440ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=202 2019-12-09T20:40:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.491+0000 7fbc027fc700 10 RGWWatcher::handle_notify() notify_id 549755813889 cookie 94398030703120 notifier 4436 bl.length()=202 2019-12-09T20:40:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.491+0000 7fbc027fc700 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2019-12-09T20:40:12.524 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.491+0000 7fbc027fc700 10 moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2019-12-09T20:40:12.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.617+0000 7fbc34440ec0 10 cache put: name=default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI info.flags=0x7 2019-12-09T20:40:12.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.617+0000 7fbc34440ec0 10 moving default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI to cache LRU end 2019-12-09T20:40:12.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.617+0000 7fbc34440ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=190 2019-12-09T20:40:12.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.628+0000 7fbc027fc700 10 RGWWatcher::handle_notify() notify_id 549755813890 cookie 94398030719952 notifier 4436 bl.length()=190 2019-12-09T20:40:12.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.628+0000 7fbc027fc700 10 cache put: name=default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI info.flags=0x7 2019-12-09T20:40:12.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.628+0000 7fbc027fc700 10 moving default.rgw.meta+users.keys+JBDGQWYZIKQZJAIXHECI to cache LRU end 2019-12-09T20:40:12.771 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2019-12-09T20:40:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "user_id": "ragweed.client.0", 2019-12-09T20:40:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "display_name": "Mr. ragweed.client.0", 2019-12-09T20:40:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "email": "ragweed.client.0+test@test.test", 2019-12-09T20:40:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "suspended": 0, 2019-12-09T20:40:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "max_buckets": 1000, 2019-12-09T20:40:12.772 INFO:teuthology.orchestra.run.smithi067.stdout: "subusers": [], 2019-12-09T20:40:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "keys": [ 2019-12-09T20:40:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-09T20:40:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "user": "ragweed.client.0", 2019-12-09T20:40:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "access_key": "JBDGQWYZIKQZJAIXHECI", 2019-12-09T20:40:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: "secret_key": "3Y9xy32//HKFJ2+tAzjg8NtuZ72tIZzokxqu277vrxn8fpItcZfazQ==" 2019-12-09T20:40:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-09T20:40:12.773 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-09T20:40:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "swift_keys": [], 2019-12-09T20:40:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": [], 2019-12-09T20:40:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "op_mask": "read, write, delete", 2019-12-09T20:40:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "system": "true", 2019-12-09T20:40:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "", 2019-12-09T20:40:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "default_storage_class": "", 2019-12-09T20:40:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_tags": [], 2019-12-09T20:40:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_quota": { 2019-12-09T20:40:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T20:40:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T20:40:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T20:40:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T20:40:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T20:40:12.775 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T20:40:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "user_quota": { 2019-12-09T20:40:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T20:40:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T20:40:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T20:40:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T20:40:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T20:40:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T20:40:12.776 INFO:teuthology.orchestra.run.smithi067.stdout: "temp_url_keys": [], 2019-12-09T20:40:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "rgw", 2019-12-09T20:40:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: "mfa_ids": [] 2019-12-09T20:40:12.777 INFO:teuthology.orchestra.run.smithi067.stdout:} 2019-12-09T20:40:12.777 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:12.837 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.804+0000 7fbc34440ec0 20 remove_watcher() i=0 2019-12-09T20:40:12.837 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.804+0000 7fbc34440ec0 2 removed watcher, disabling cache 2019-12-09T20:40:12.880 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.847+0000 7fbc34440ec0 20 remove_watcher() i=1 2019-12-09T20:40:12.940 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.906+0000 7fbc34440ec0 20 remove_watcher() i=2 2019-12-09T20:40:13.007 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:12.974+0000 7fbc34440ec0 20 remove_watcher() i=3 2019-12-09T20:40:13.061 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:13.028+0000 7fbc34440ec0 20 remove_watcher() i=4 2019-12-09T20:40:13.113 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:13.080+0000 7fbc34440ec0 20 remove_watcher() i=5 2019-12-09T20:40:13.165 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:13.132+0000 7fbc34440ec0 20 remove_watcher() i=6 2019-12-09T20:40:13.217 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:40:13.183+0000 7fbc34440ec0 20 remove_watcher() i=7 2019-12-09T20:40:13.226 INFO:tasks.ragweed:Configuring ragweed... 2019-12-09T20:40:13.226 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest/ragweed && ./bootstrap 2019-12-09T20:40:19.380 INFO:teuthology.orchestra.run.smithi067.stderr:./bootstrap: missing required RPM packages. Installing via sudo. 2019-12-09T20:40:19.485 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T20:40:19.523 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T20:40:19.524 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T20:40:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T20:40:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T20:40:19.525 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T20:40:20.196 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T20:40:20.592 INFO:teuthology.orchestra.run.smithi067.stdout:Package python2-pip-8.1.2-10.el7.noarch already installed and latest version 2019-12-09T20:40:21.482 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T20:40:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:40:21.483 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libevent-devel.x86_64 0:2.0.21-4.el7 will be installed 2019-12-09T20:40:21.485 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libxml2-devel.x86_64 0:2.9.1-6.el7_2.3 will be installed 2019-12-09T20:40:21.487 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: xz-devel for package: libxml2-devel-2.9.1-6.el7_2.3.x86_64 2019-12-09T20:40:21.521 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libxslt-devel.x86_64 0:1.1.28-5.el7 will be installed 2019-12-09T20:40:21.522 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libgcrypt-devel for package: libxslt-devel-1.1.28-5.el7.x86_64 2019-12-09T20:40:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed 2019-12-09T20:40:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:40:21.525 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libgcrypt-devel.x86_64 0:1.5.3-14.el7 will be installed 2019-12-09T20:40:21.527 INFO:teuthology.orchestra.run.smithi067.stdout:--> Processing Dependency: libgpg-error-devel for package: libgcrypt-devel-1.5.3-14.el7.x86_64 2019-12-09T20:40:21.530 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package xz-devel.x86_64 0:5.2.2-1.el7 will be installed 2019-12-09T20:40:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T20:40:21.531 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libgpg-error-devel.x86_64 0:1.12-3.el7 will be installed 2019-12-09T20:40:21.924 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T20:40:22.117 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:22.118 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T20:40:22.119 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:22.119 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:40:22.120 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2019-12-09T20:40:22.120 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:40:22.120 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T20:40:22.120 INFO:teuthology.orchestra.run.smithi067.stdout: libevent-devel x86_64 2.0.21-4.el7 base 85 k 2019-12-09T20:40:22.121 INFO:teuthology.orchestra.run.smithi067.stdout: libxml2-devel x86_64 2.9.1-6.el7_2.3 base 1.0 M 2019-12-09T20:40:22.121 INFO:teuthology.orchestra.run.smithi067.stdout: libxslt-devel x86_64 1.1.28-5.el7 base 309 k 2019-12-09T20:40:22.121 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.7-18.el7 base 50 k 2019-12-09T20:40:22.121 INFO:teuthology.orchestra.run.smithi067.stdout:Installing for dependencies: 2019-12-09T20:40:22.121 INFO:teuthology.orchestra.run.smithi067.stdout: libgcrypt-devel x86_64 1.5.3-14.el7 base 129 k 2019-12-09T20:40:22.122 INFO:teuthology.orchestra.run.smithi067.stdout: libgpg-error-devel x86_64 1.12-3.el7 base 16 k 2019-12-09T20:40:22.122 INFO:teuthology.orchestra.run.smithi067.stdout: xz-devel x86_64 5.2.2-1.el7 base 46 k 2019-12-09T20:40:22.122 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:22.122 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T20:40:22.122 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T20:40:22.123 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages (+3 Dependent packages) 2019-12-09T20:40:22.123 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:22.123 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 1.7 M 2019-12-09T20:40:22.123 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 12 M 2019-12-09T20:40:22.123 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T20:40:22.341 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2019-12-09T20:40:22.342 INFO:teuthology.orchestra.run.smithi067.stdout:Total 7.5 MB/s | 1.7 MB 00:00 2019-12-09T20:40:22.347 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T20:40:22.356 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T20:40:22.404 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T20:40:22.405 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T20:40:22.992 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgpg-error-devel-1.12-3.el7.x86_64 1/7 2019-12-09T20:40:23.264 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgcrypt-devel-1.5.3-14.el7.x86_64 2/7 2019-12-09T20:40:23.518 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.7-18.el7.x86_64 3/7 2019-12-09T20:40:24.061 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xz-devel-5.2.2-1.el7.x86_64 4/7 2019-12-09T20:40:24.497 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libxml2-devel-2.9.1-6.el7_2.3.x86_64 5/7 2019-12-09T20:40:24.847 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libxslt-devel-1.1.28-5.el7.x86_64 6/7 2019-12-09T20:40:25.109 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libevent-devel-2.0.21-4.el7.x86_64 7/7 2019-12-09T20:40:25.217 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xz-devel-5.2.2-1.el7.x86_64 1/7 2019-12-09T20:40:25.317 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libxml2-devel-2.9.1-6.el7_2.3.x86_64 2/7 2019-12-09T20:40:25.409 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.7-18.el7.x86_64 3/7 2019-12-09T20:40:25.529 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libxslt-devel-1.1.28-5.el7.x86_64 4/7 2019-12-09T20:40:25.644 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgpg-error-devel-1.12-3.el7.x86_64 5/7 2019-12-09T20:40:25.753 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libevent-devel-2.0.21-4.el7.x86_64 6/7 2019-12-09T20:40:26.245 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgcrypt-devel-1.5.3-14.el7.x86_64 7/7 2019-12-09T20:40:26.246 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:26.246 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T20:40:26.246 INFO:teuthology.orchestra.run.smithi067.stdout: libevent-devel.x86_64 0:2.0.21-4.el7 libxml2-devel.x86_64 0:2.9.1-6.el7_2.3 2019-12-09T20:40:26.246 INFO:teuthology.orchestra.run.smithi067.stdout: libxslt-devel.x86_64 0:1.1.28-5.el7 zlib-devel.x86_64 0:1.2.7-18.el7 2019-12-09T20:40:26.246 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:26.247 INFO:teuthology.orchestra.run.smithi067.stdout:Dependency Installed: 2019-12-09T20:40:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: libgcrypt-devel.x86_64 0:1.5.3-14.el7 libgpg-error-devel.x86_64 0:1.12-3.el7 2019-12-09T20:40:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: xz-devel.x86_64 0:5.2.2-1.el7 2019-12-09T20:40:26.247 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:26.247 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T20:40:26.441 INFO:teuthology.orchestra.run.smithi067.stdout:New python executable in /home/ubuntu/cephtest/ragweed/virtualenv/bin/python 2019-12-09T20:40:28.267 INFO:teuthology.orchestra.run.smithi067.stdout:Installing setuptools, pip, wheel...done. 2019-12-09T20:40:28.771 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting pip 2019-12-09T20:40:28.865 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB) 2019-12-09T20:40:29.073 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: pip 2019-12-09T20:40:29.073 INFO:teuthology.orchestra.run.smithi067.stdout: Found existing installation: pip 9.0.1 2019-12-09T20:40:29.178 INFO:teuthology.orchestra.run.smithi067.stdout: Uninstalling pip-9.0.1: 2019-12-09T20:40:29.256 INFO:teuthology.orchestra.run.smithi067.stdout: Successfully uninstalled pip-9.0.1 2019-12-09T20:40:29.961 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed pip-19.3.1 2019-12-09T20:40:30.783 INFO:teuthology.orchestra.run.smithi067.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-09T20:40:30.867 INFO:teuthology.orchestra.run.smithi067.stdout:Requirement already satisfied: six in /usr/lib/python2.7/site-packages (1.9.0) 2019-12-09T20:40:31.994 INFO:teuthology.orchestra.run.smithi067.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-09T20:40:32.181 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting nose 2019-12-09T20:40:32.220 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl (154kB) 2019-12-09T20:40:32.347 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: nose 2019-12-09T20:40:32.421 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed nose-1.3.7 2019-12-09T20:40:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-09T20:40:33.559 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting setuptools==32.3.1 2019-12-09T20:40:33.602 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/69/19/b1dff551058ce79d88b1e3688f1c735590d7ddf44d10681512133b35019f/setuptools-32.3.1-py2.py3-none-any.whl (479kB) 2019-12-09T20:40:33.919 INFO:teuthology.orchestra.run.smithi067.stderr:ERROR: kubernetes 10.0.1 has requirement ipaddress>=1.0.17, but you'll have ipaddress 1.0.16 which is incompatible. 2019-12-09T20:40:33.920 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: setuptools 2019-12-09T20:40:33.920 INFO:teuthology.orchestra.run.smithi067.stdout: Found existing installation: setuptools 28.8.0 2019-12-09T20:40:33.939 INFO:teuthology.orchestra.run.smithi067.stdout: Uninstalling setuptools-28.8.0: 2019-12-09T20:40:33.950 INFO:teuthology.orchestra.run.smithi067.stdout: Successfully uninstalled setuptools-28.8.0 2019-12-09T20:40:34.089 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed setuptools-32.3.1 2019-12-09T20:40:34.684 INFO:teuthology.orchestra.run.smithi067.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-09T20:40:34.998 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting PyYAML 2019-12-09T20:40:35.050 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/8d/c9/e5be955a117a1ac548cdd31e37e8fd7b02ce987f9655f5c7563c656d5dcb/PyYAML-5.2.tar.gz (265kB) 2019-12-09T20:40:35.717 INFO:teuthology.orchestra.run.smithi067.stdout:Requirement already up-to-date: nose>=1.0.0 in ./virtualenv/lib/python2.7/site-packages (from -r requirements.txt (line 2)) (1.3.7) 2019-12-09T20:40:35.873 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting boto!=2.46.0,>=2.6.0 2019-12-09T20:40:35.885 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl (1.4MB) 2019-12-09T20:40:36.221 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting bunch>=1.0.0 2019-12-09T20:40:36.232 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/ef/bf/a4cf1779a4ffb4f610903fa08e15d1f4a8a2f4e3353a02afbe097c5bf4a8/bunch-1.0.1.tar.gz 2019-12-09T20:40:36.919 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting gevent>=1.0 2019-12-09T20:40:36.934 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/e6/e9/3a693414f706e12abe60554cd73c5ae8f848b182ae58018f93d86c9eb418/gevent-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl (5.0MB) 2019-12-09T20:40:37.455 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting httplib2 2019-12-09T20:40:37.466 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/ce/2e/87461bfbb7e561203b759b3f7f639e2144226604372830d00a8279960ae1/httplib2-0.14.0.tar.gz (219kB) 2019-12-09T20:40:38.741 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting lxml 2019-12-09T20:40:38.758 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/25/73/730ad7249847f741c7e622f52b971daa5540f6fb87589bc92c717f5aafba/lxml-4.4.2-cp27-cp27mu-manylinux1_x86_64.whl (5.7MB) 2019-12-09T20:40:39.295 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting greenlet>=0.4.14; platform_python_implementation == "CPython" 2019-12-09T20:40:39.307 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/8b/6e/f2d25875713ad0885c8d3c69269697406652e6f64e1a6bd8264f7a609327/greenlet-0.4.15-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-09T20:40:39.396 INFO:teuthology.orchestra.run.smithi067.stdout:Building wheels for collected packages: PyYAML, bunch, httplib2 2019-12-09T20:40:39.396 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for PyYAML (setup.py): started 2019-12-09T20:40:40.049 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for PyYAML (setup.py): finished with status 'done' 2019-12-09T20:40:40.049 INFO:teuthology.orchestra.run.smithi067.stdout: Created wheel for PyYAML: filename=PyYAML-5.2-cp27-cp27mu-linux_x86_64.whl size=46299 sha256=a1c1c7d6e13868902abb3d2bee49aebfa46115f56e211f4770ae466a48a871cc 2019-12-09T20:40:40.050 INFO:teuthology.orchestra.run.smithi067.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/54/b7/c7/2ada654ee54483c9329871665aaf4a6056c3ce36f29cf66e67 2019-12-09T20:40:40.066 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for bunch (setup.py): started 2019-12-09T20:40:40.445 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for bunch (setup.py): finished with status 'done' 2019-12-09T20:40:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: Created wheel for bunch: filename=bunch-1.0.1-cp27-none-any.whl size=8347 sha256=c033171ef8d196712f8aaf3a9313378c4f7ccf3d03357e3027b8e793f0510363 2019-12-09T20:40:40.446 INFO:teuthology.orchestra.run.smithi067.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/56/0f/19/fbbf81e5764e6d8b74501c4357a88c14c94466ec777c03734c 2019-12-09T20:40:40.449 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for httplib2 (setup.py): started 2019-12-09T20:40:40.793 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for httplib2 (setup.py): finished with status 'done' 2019-12-09T20:40:40.795 INFO:teuthology.orchestra.run.smithi067.stdout: Created wheel for httplib2: filename=httplib2-0.14.0-cp27-none-any.whl size=96742 sha256=aa567d462f2bb0f6baa440b7a34d61784f310f320a0b0b6ecef8526b87911b87 2019-12-09T20:40:40.796 INFO:teuthology.orchestra.run.smithi067.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/51/ab/65/f318b4271019823cf7d7c178ac76a8ea4d6e6271fe9f5ba488 2019-12-09T20:40:40.800 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully built PyYAML bunch httplib2 2019-12-09T20:40:40.912 INFO:teuthology.orchestra.run.smithi067.stderr:ERROR: kubernetes 10.0.1 has requirement ipaddress>=1.0.17, but you'll have ipaddress 1.0.16 which is incompatible. 2019-12-09T20:40:40.913 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: PyYAML, boto, bunch, greenlet, gevent, httplib2, lxml 2019-12-09T20:40:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: Found existing installation: PyYAML 3.10 2019-12-09T20:40:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: Not uninstalling pyyaml at /usr/lib64/python2.7/site-packages, outside environment /home/ubuntu/cephtest/ragweed/virtualenv 2019-12-09T20:40:40.913 INFO:teuthology.orchestra.run.smithi067.stdout: Can't uninstall 'PyYAML'. No files were found to uninstall. 2019-12-09T20:40:41.983 INFO:teuthology.orchestra.run.smithi067.stdout: Found existing installation: lxml 3.2.1 2019-12-09T20:40:41.983 INFO:teuthology.orchestra.run.smithi067.stdout: Not uninstalling lxml at /usr/lib64/python2.7/site-packages, outside environment /home/ubuntu/cephtest/ragweed/virtualenv 2019-12-09T20:40:41.983 INFO:teuthology.orchestra.run.smithi067.stdout: Can't uninstall 'lxml'. No files were found to uninstall. 2019-12-09T20:40:42.087 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed PyYAML-5.2 boto-2.49.0 bunch-1.0.1 gevent-1.4.0 greenlet-0.4.15 httplib2-0.14.0 lxml-4.4.2 2019-12-09T20:40:42.469 INFO:teuthology.orchestra.run.smithi067.stdout:running develop 2019-12-09T20:40:42.536 INFO:teuthology.orchestra.run.smithi067.stdout:running egg_info 2019-12-09T20:40:42.536 INFO:teuthology.orchestra.run.smithi067.stdout:creating ragweed.egg-info 2019-12-09T20:40:42.537 INFO:teuthology.orchestra.run.smithi067.stdout:writing requirements to ragweed.egg-info/requires.txt 2019-12-09T20:40:42.537 INFO:teuthology.orchestra.run.smithi067.stdout:writing ragweed.egg-info/PKG-INFO 2019-12-09T20:40:42.537 INFO:teuthology.orchestra.run.smithi067.stdout:writing top-level names to ragweed.egg-info/top_level.txt 2019-12-09T20:40:42.537 INFO:teuthology.orchestra.run.smithi067.stdout:writing dependency_links to ragweed.egg-info/dependency_links.txt 2019-12-09T20:40:42.537 INFO:teuthology.orchestra.run.smithi067.stdout:writing manifest file 'ragweed.egg-info/SOURCES.txt' 2019-12-09T20:40:42.539 INFO:teuthology.orchestra.run.smithi067.stdout:reading manifest file 'ragweed.egg-info/SOURCES.txt' 2019-12-09T20:40:42.539 INFO:teuthology.orchestra.run.smithi067.stdout:writing manifest file 'ragweed.egg-info/SOURCES.txt' 2019-12-09T20:40:42.539 INFO:teuthology.orchestra.run.smithi067.stdout:running build_ext 2019-12-09T20:40:42.539 INFO:teuthology.orchestra.run.smithi067.stdout:Creating /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages/ragweed.egg-link (link to .) 2019-12-09T20:40:42.539 INFO:teuthology.orchestra.run.smithi067.stdout:Adding ragweed 0.0.1 to easy-install.pth file 2019-12-09T20:40:42.540 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:42.540 INFO:teuthology.orchestra.run.smithi067.stdout:Installed /home/ubuntu/cephtest/ragweed 2019-12-09T20:40:42.541 INFO:teuthology.orchestra.run.smithi067.stdout:Processing dependencies for ragweed==0.0.1 2019-12-09T20:40:42.543 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for isodate>=0.4.4 2019-12-09T20:40:42.543 INFO:teuthology.orchestra.run.smithi067.stdout:Reading https://pypi.python.org/simple/isodate/ 2019-12-09T20:40:42.646 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading https://files.pythonhosted.org/packages/b1/80/fb8c13a4cd38eb5021dc3741a9e588e4d1de88d895c1910c6fc8a08b7a70/isodate-0.6.0.tar.gz#sha256=2e364a3d5759479cdb2d37cce6b9376ea504db2ff90252a2e5b7cc89cc9ff2d8 2019-12-09T20:40:42.696 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: isodate 0.6.0 2019-12-09T20:40:42.697 INFO:teuthology.orchestra.run.smithi067.stdout:Processing isodate-0.6.0.tar.gz 2019-12-09T20:40:42.706 INFO:teuthology.orchestra.run.smithi067.stdout:Writing /tmp/easy_install-3S5MS3/isodate-0.6.0/setup.cfg 2019-12-09T20:40:42.706 INFO:teuthology.orchestra.run.smithi067.stdout:Running isodate-0.6.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3S5MS3/isodate-0.6.0/egg-dist-tmp-VpBNRy 2019-12-09T20:40:42.996 INFO:teuthology.orchestra.run.smithi067.stderr:zip_safe flag not set; analyzing archive contents... 2019-12-09T20:40:43.012 INFO:teuthology.orchestra.run.smithi067.stdout:Moving isodate-0.6.0-py2.7.egg to /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-09T20:40:43.013 INFO:teuthology.orchestra.run.smithi067.stdout:Adding isodate 0.6.0 to easy-install.pth file 2019-12-09T20:40:43.014 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:43.014 INFO:teuthology.orchestra.run.smithi067.stdout:Installed /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages/isodate-0.6.0-py2.7.egg 2019-12-09T20:40:43.038 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for gevent==1.4.0 2019-12-09T20:40:43.039 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: gevent 1.4.0 2019-12-09T20:40:43.039 INFO:teuthology.orchestra.run.smithi067.stdout:Adding gevent 1.4.0 to easy-install.pth file 2019-12-09T20:40:43.039 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:43.039 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-09T20:40:43.040 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for bunch==1.0.1 2019-12-09T20:40:43.041 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: bunch 1.0.1 2019-12-09T20:40:43.041 INFO:teuthology.orchestra.run.smithi067.stdout:Adding bunch 1.0.1 to easy-install.pth file 2019-12-09T20:40:43.041 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:43.041 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-09T20:40:43.043 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for PyYAML==5.2 2019-12-09T20:40:43.043 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: PyYAML 5.2 2019-12-09T20:40:43.044 INFO:teuthology.orchestra.run.smithi067.stdout:Adding PyYAML 5.2 to easy-install.pth file 2019-12-09T20:40:43.044 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:43.044 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-09T20:40:43.045 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for boto==2.49.0 2019-12-09T20:40:43.045 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: boto 2.49.0 2019-12-09T20:40:43.045 INFO:teuthology.orchestra.run.smithi067.stdout:Adding boto 2.49.0 to easy-install.pth file 2019-12-09T20:40:43.045 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:43.046 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-09T20:40:43.047 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for six==1.9.0 2019-12-09T20:40:43.047 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: six 1.9.0 2019-12-09T20:40:43.047 INFO:teuthology.orchestra.run.smithi067.stdout:Adding six 1.9.0 to easy-install.pth file 2019-12-09T20:40:43.047 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:43.048 INFO:teuthology.orchestra.run.smithi067.stdout:Using /usr/lib/python2.7/site-packages 2019-12-09T20:40:43.058 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for greenlet==0.4.15 2019-12-09T20:40:43.058 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: greenlet 0.4.15 2019-12-09T20:40:43.058 INFO:teuthology.orchestra.run.smithi067.stdout:Adding greenlet 0.4.15 to easy-install.pth file 2019-12-09T20:40:43.059 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:40:43.059 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/ragweed/virtualenv/lib/python2.7/site-packages 2019-12-09T20:40:43.059 INFO:teuthology.orchestra.run.smithi067.stdout:Finished processing dependencies for ragweed==0.0.1 2019-12-09T20:40:43.071 INFO:teuthology.orchestra.run.smithi067:> cat > /home/ubuntu/cephtest/archive/ragweed.client.0.conf 2019-12-09T20:40:43.118 INFO:tasks.ragweed:Configuring boto... 2019-12-09T20:40:43.118 INFO:teuthology.orchestra.run.smithi067:> cat > /home/ubuntu/cephtest/boto.cfg 2019-12-09T20:40:43.215 INFO:teuthology.orchestra.run.smithi067:ragweed tests against rgw> RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw' 2019-12-09T20:41:17.033 INFO:teuthology.orchestra.run.smithi067.stderr:ragweed.tests.tests.r_test_multipart_defer_complete.test ... ok 2019-12-09T20:41:45.460 INFO:teuthology.orchestra.run.smithi067.stderr:ragweed.tests.tests.r_test_multipart_defer_update_complete.test ... ok 2019-12-09T20:42:08.660 INFO:teuthology.orchestra.run.smithi067.stderr:ragweed.tests.tests.r_test_multipart_index_versioning.test ... ok 2019-12-09T20:42:34.240 INFO:teuthology.orchestra.run.smithi067.stderr:ragweed.tests.tests.r_test_multipart_simple.test ... ok 2019-12-09T20:42:45.733 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:06.837 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:42:45.794 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:06.898 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:42:45.795 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:06.966 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:42:45.817 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:06.921 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:42:45.840 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:07.011 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:42:45.864 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:06.968 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:42:46.432 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:09:07.604 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:42:46.548 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:07.653 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:42:46.570 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:07.675 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:43:03.687 INFO:teuthology.orchestra.run.smithi067.stderr:ragweed.tests.tests.r_test_obj_storage_class.test ... ok 2019-12-09T20:44:12.933 INFO:teuthology.orchestra.run.smithi067.stderr:ragweed.tests.tests.r_test_obj_storage_class_copy.test ... ok 2019-12-09T20:44:47.323 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:08.419 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:44:47.328 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:08.424 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:44:47.337 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:08.430 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:44:47.340 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:08.512 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:44:47.341 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:08.437 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:44:47.344 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:08.518 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:44:47.448 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:08.544 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:44:47.458 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:11:08.631 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:44:47.465 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:08.561 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:45:08.984 INFO:teuthology.orchestra.run.smithi067.stderr:ragweed.tests.tests.r_test_obj_storage_class_multipart.test ... ok 2019-12-09T20:46:46.917 INFO:teuthology.orchestra.run.smithi067.stderr:ragweed.tests.tests.r_test_obj_storage_class_multipart_copy.test ... ok 2019-12-09T20:47:20.706 INFO:teuthology.orchestra.run.smithi067.stderr:ragweed.tests.tests.r_test_small_obj_data.test ... ok 2019-12-09T20:47:20.707 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T20:47:20.707 INFO:teuthology.orchestra.run.smithi067.stderr:---------------------------------------------------------------------- 2019-12-09T20:47:20.707 INFO:teuthology.orchestra.run.smithi067.stderr:Ran 9 tests in 397.087s 2019-12-09T20:47:20.707 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T20:47:20.708 INFO:teuthology.orchestra.run.smithi067.stderr:OK 2019-12-09T20:47:20.729 INFO:tasks.ragweed:Cleaning up boto... 2019-12-09T20:47:20.729 INFO:teuthology.orchestra.run.smithi067:> rm /home/ubuntu/cephtest/boto.cfg 2019-12-09T20:47:20.784 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid sysuser.client.0 --purge-data 2019-12-09T20:47:20.903 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:20.841+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:20.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:20.864+0000 7fa704dafec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:20.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:20.864+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:20.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:20.873+0000 7fa704dafec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:20.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:20.873+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:20.878+0000 7fa704dafec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:20.878+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:20.952 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:20.890+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:21.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.078+0000 7fa704dafec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T20:47:21.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.078+0000 7fa704dafec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T20:47:21.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.078+0000 7fa704dafec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T20:47:21.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.078+0000 7fa704dafec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T20:47:21.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.078+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.148 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.085+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:21.148 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.085+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.157 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.095+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:47:21.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.095+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.103+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:21.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.103+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.110+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:47:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.110+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.119+0000 7fa704dafec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.119+0000 7fa704dafec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T20:47:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.119+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.126+0000 7fa704dafec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:21.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.126+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.137+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:21.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.137+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.149+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.149+0000 7fa704dafec0 20 zonegroup default 2019-12-09T20:47:21.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.149+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.155+0000 7fa704dafec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.155+0000 7fa704dafec0 10 Cannot find current period zone using local zone 2019-12-09T20:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.155+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.160+0000 7fa704dafec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:21.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.160+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.168+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:21.230 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.168+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:21.239 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.176+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:47:21.239 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.176+0000 7fa704dafec0 20 zone default found 2019-12-09T20:47:21.239 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.176+0000 7fa704dafec0 20 started sync module instance, tier type = 2019-12-09T20:47:21.239 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.176+0000 7fa704dafec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T20:47:21.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.558+0000 7fa704dafec0 20 add_watcher() i=0 2019-12-09T20:47:21.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.558+0000 7fa704dafec0 20 add_watcher() i=1 2019-12-09T20:47:21.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.558+0000 7fa704dafec0 20 add_watcher() i=2 2019-12-09T20:47:21.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.558+0000 7fa704dafec0 20 add_watcher() i=3 2019-12-09T20:47:21.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.558+0000 7fa704dafec0 20 add_watcher() i=4 2019-12-09T20:47:21.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.558+0000 7fa704dafec0 20 add_watcher() i=5 2019-12-09T20:47:21.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.581+0000 7fa704dafec0 20 add_watcher() i=6 2019-12-09T20:47:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.588+0000 7fa704dafec0 20 add_watcher() i=7 2019-12-09T20:47:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.588+0000 7fa704dafec0 2 all 8 watchers are set, enabling cache 2019-12-09T20:47:21.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.589+0000 7fa6c27dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:47:21.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.595+0000 7fa6f1fd9700 0 -- 172.21.15.67:0/3089597153 >> v2:172.21.15.81:3300/0 conn(0x55cba6c07fc0 msgr2=0x55cba6c0e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T20:47:21.860 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.797+0000 7fa6f1fd9700 0 --2- 172.21.15.67:0/3089597153 >> v2:172.21.15.81:3300/0 conn(0x55cba6c07fc0 0x55cba6c0e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-09T20:47:21.929 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:21.866+0000 7fa6f27da700 0 -- 172.21.15.67:0/4152726196 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55cba6c0dbb0 msgr2=0x55cba6c0dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:47:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.369+0000 7fa638ff9700 20 reqs_thread_entry: start 2019-12-09T20:47:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.369+0000 7fa6327fc700 20 reqs_thread_entry: start 2019-12-09T20:47:23.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.369+0000 7fa704dafec0 20 init_complete bucket index max shards: 0 2019-12-09T20:47:23.433 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.370+0000 7fa630ff9700 20 reqs_thread_entry: start 2019-12-09T20:47:23.433 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.370+0000 7fa704dafec0 20 get_system_obj_state: rctx=0x7ffcd7797958 obj=default.rgw.meta:users.uid:sysuser.client.0 state=0x55cba6c88c90 s->prefetch_data=0 2019-12-09T20:47:23.433 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.370+0000 7fa704dafec0 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2019-12-09T20:47:23.442 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.379+0000 7fa704dafec0 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x16 2019-12-09T20:47:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.379+0000 7fa704dafec0 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-09T20:47:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.379+0000 7fa704dafec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.379+0000 7fa704dafec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.379+0000 7fa704dafec0 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-09T20:47:23.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.380+0000 7fa704dafec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.392+0000 7fa704dafec0 20 rados_obj.operate() r=0 bl.length=448 2019-12-09T20:47:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.392+0000 7fa704dafec0 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x11 2019-12-09T20:47:23.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.392+0000 7fa704dafec0 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2019-12-09T20:47:23.468 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.405+0000 7fa704dafec0 10 removing key index: QHAWDUANSLCTQKJUAZPK 2019-12-09T20:47:23.468 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.405+0000 7fa704dafec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=170 2019-12-09T20:47:23.502 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.439+0000 7fa6d2ffd700 10 RGWWatcher::handle_notify() notify_id 592705486849 cookie 94333164261120 notifier 4457 bl.length()=170 2019-12-09T20:47:23.928 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.865+0000 7fa704dafec0 10 removing email index: sysuser.client.0+test@test.test 2019-12-09T20:47:23.928 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.865+0000 7fa704dafec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=182 2019-12-09T20:47:23.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:23.876+0000 7fa6d2ffd700 10 RGWWatcher::handle_notify() notify_id 592705486851 cookie 94333164270128 notifier 4457 bl.length()=182 2019-12-09T20:47:24.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.183+0000 7fa704dafec0 10 removing user buckets index 2019-12-09T20:47:24.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.183+0000 7fa704dafec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=173 2019-12-09T20:47:24.276 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.213+0000 7fa6d2ffd700 10 RGWWatcher::handle_notify() notify_id 592705486848 cookie 94333164268336 notifier 4457 bl.length()=173 2019-12-09T20:47:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.378+0000 7fa704dafec0 10 removing user index: sysuser.client.0 2019-12-09T20:47:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.378+0000 7fa704dafec0 10 removing default.rgw.meta+users.uid+sysuser.client.0 from cache 2019-12-09T20:47:24.441 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.378+0000 7fa704dafec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=165 2019-12-09T20:47:24.452 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.389+0000 7fa6d2ffd700 10 RGWWatcher::handle_notify() notify_id 592705486850 cookie 94333164267024 notifier 4457 bl.length()=165 2019-12-09T20:47:24.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.550+0000 7fa704dafec0 20 remove_watcher() i=0 2019-12-09T20:47:24.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.551+0000 7fa704dafec0 2 removed watcher, disabling cache 2019-12-09T20:47:24.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.611+0000 7fa704dafec0 20 remove_watcher() i=1 2019-12-09T20:47:24.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.649+0000 7fa704dafec0 20 remove_watcher() i=2 2019-12-09T20:47:24.749 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.686+0000 7fa704dafec0 20 remove_watcher() i=3 2019-12-09T20:47:24.814 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.752+0000 7fa704dafec0 20 remove_watcher() i=4 2019-12-09T20:47:24.849 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.787+0000 7fa704dafec0 20 remove_watcher() i=5 2019-12-09T20:47:24.882 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.820+0000 7fa704dafec0 20 remove_watcher() i=6 2019-12-09T20:47:24.917 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.854+0000 7fa704dafec0 20 remove_watcher() i=7 2019-12-09T20:47:24.924 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid ragweed.client.0 --purge-data 2019-12-09T20:47:25.045 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:24.982+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.078 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.015+0000 7f63a9625ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:25.078 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.015+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.091 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.028+0000 7f63a9625ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:25.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.028+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.098 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.035+0000 7f63a9625ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:25.098 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.035+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.108 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.045+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:25.278 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.214+0000 7f63a9625ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T20:47:25.278 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.214+0000 7f63a9625ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T20:47:25.278 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.214+0000 7f63a9625ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T20:47:25.278 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.214+0000 7f63a9625ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T20:47:25.278 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.214+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.223+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:25.286 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.223+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.234+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:47:25.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.235+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.246+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:25.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.246+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.317 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.254+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:47:25.317 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.254+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.337 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.260+0000 7f63a9625ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:25.337 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.260+0000 7f63a9625ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T20:47:25.338 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.260+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.338 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.266+0000 7f63a9625ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:25.338 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.266+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.338 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.274+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:25.338 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.274+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.282+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:47:25.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.282+0000 7f63a9625ec0 20 zonegroup default 2019-12-09T20:47:25.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.282+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.287+0000 7f63a9625ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.287+0000 7f63a9625ec0 10 Cannot find current period zone using local zone 2019-12-09T20:47:25.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.287+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.354 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.292+0000 7f63a9625ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:47:25.354 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.292+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.302+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:47:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.302+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:25.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.311+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:47:25.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.311+0000 7f63a9625ec0 20 zone default found 2019-12-09T20:47:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.311+0000 7f63a9625ec0 20 started sync module instance, tier type = 2019-12-09T20:47:25.375 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.311+0000 7f63a9625ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T20:47:25.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.684+0000 7f63a9625ec0 20 add_watcher() i=0 2019-12-09T20:47:25.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.684+0000 7f63a9625ec0 20 add_watcher() i=1 2019-12-09T20:47:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.684+0000 7f63a9625ec0 20 add_watcher() i=2 2019-12-09T20:47:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.684+0000 7f63a9625ec0 20 add_watcher() i=3 2019-12-09T20:47:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.684+0000 7f63a9625ec0 20 add_watcher() i=4 2019-12-09T20:47:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.684+0000 7f63a9625ec0 20 add_watcher() i=5 2019-12-09T20:47:25.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.684+0000 7f63a9625ec0 20 add_watcher() i=6 2019-12-09T20:47:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.710+0000 7f63a9625ec0 20 add_watcher() i=7 2019-12-09T20:47:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.710+0000 7f63a9625ec0 2 all 8 watchers are set, enabling cache 2019-12-09T20:47:25.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.710+0000 7f63677de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:47:25.777 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.713+0000 7f639604e700 0 -- 172.21.15.67:0/2292498524 >> v1:172.21.15.81:6789/0 conn(0x55d322baebe0 legacy=0x55d322bb1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:47:25.812 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:25.749+0000 7f6397050700 0 -- 172.21.15.67:0/746137772 >> [v2:172.21.15.67:6800/9311,v1:172.21.15.67:6801/9311] conn(0x7f62d80224c0 msgr2=0x7f62d8024990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:47:27.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.086+0000 7f62d5ffb700 20 reqs_thread_entry: start 2019-12-09T20:47:27.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.087+0000 7f62c77fe700 20 reqs_thread_entry: start 2019-12-09T20:47:27.151 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.087+0000 7f63a9625ec0 20 init_complete bucket index max shards: 0 2019-12-09T20:47:27.151 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.087+0000 7f62c5ffb700 20 reqs_thread_entry: start 2019-12-09T20:47:27.151 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.087+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb3828 obj=default.rgw.meta:users.uid:ragweed.client.0 state=0x55d322c2d8d0 s->prefetch_data=0 2019-12-09T20:47:27.151 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.087+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2019-12-09T20:47:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.100+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x16 2019-12-09T20:47:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.100+0000 7f63a9625ec0 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-09T20:47:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.100+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.100+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.100+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-09T20:47:27.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.100+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.115+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=448 2019-12-09T20:47:27.178 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.115+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x11 2019-12-09T20:47:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.115+0000 7f63a9625ec0 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2019-12-09T20:47:27.237 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.173+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-conf state=0x55d322c2f1f0 s->prefetch_data=0 2019-12-09T20:47:27.238 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.173+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-conf : miss 2019-12-09T20:47:27.252 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.189+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-conf info.flags=0x6 2019-12-09T20:47:27.253 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.189+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-conf to cache LRU end 2019-12-09T20:47:27.253 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.189+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:27.253 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.189+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:27.254 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.189+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-conf : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:27.254 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.189+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.216+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=200 2019-12-09T20:47:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.216+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-conf info.flags=0x1 2019-12-09T20:47:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.216+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-conf to cache LRU end 2019-12-09T20:47:27.279 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.216+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-conf:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 state=0x55d322c317c0 s->prefetch_data=0 2019-12-09T20:47:27.280 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.216+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 : miss 2019-12-09T20:47:27.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.245+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 info.flags=0x16 2019-12-09T20:47:27.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.245+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-conf:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 to cache LRU end 2019-12-09T20:47:27.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.245+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:27.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.245+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:27.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.245+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:27.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.245+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:27.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.245+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:27.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.245+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:27.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.245+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:27.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.278+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=292 2019-12-09T20:47:27.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.278+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 info.flags=0x13 2019-12-09T20:47:27.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.278+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-conf:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 to cache LRU end 2019-12-09T20:47:27.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.278+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:27.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.278+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:27.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.278+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-conf:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:27.363 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.300+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-conf[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1]) start [] num_entries 1100 2019-12-09T20:47:27.389 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_multipart_defer_complete[] 2019-12-09T20:47:27.389 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_multipart_defer_update_complete[] 2019-12-09T20:47:27.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_multipart_index_versioning[] 2019-12-09T20:47:27.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_multipart_simple[] 2019-12-09T20:47:27.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_obj_storage_class[] 2019-12-09T20:47:27.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_obj_storage_class_copy[] 2019-12-09T20:47:27.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart[] 2019-12-09T20:47:27.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart_copy[] 2019-12-09T20:47:27.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got tests/r_test_small_obj_data[] 2019-12-09T20:47:27.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.326+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:27.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.340+0000 7f63a9625ec0 10 manifest: total_size = 3872 2019-12-09T20:47:27.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.340+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.13 2019-12-09T20:47:27.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.340+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:27.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.341+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c369a8 2019-12-09T20:47:27.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.341+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:27.580 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.517+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:27.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.533+0000 7f63a9625ec0 10 manifest: total_size = 3881 2019-12-09T20:47:27.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.533+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.30 2019-12-09T20:47:27.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.533+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:27.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.533+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c369a8 2019-12-09T20:47:27.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.533+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:27.764 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.700+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:27.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.733+0000 7f63a9625ec0 10 manifest: total_size = 9698 2019-12-09T20:47:27.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.733+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.56 2019-12-09T20:47:27.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.733+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:27.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.733+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c369a8 2019-12-09T20:47:27.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.733+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:27.955 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.891+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:27.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.907+0000 7f63a9625ec0 10 manifest: total_size = 9701 2019-12-09T20:47:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.907+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.72 2019-12-09T20:47:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.907+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.907+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c369a8 2019-12-09T20:47:27.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:27.907+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:28.237 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.173+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.207+0000 7f63a9625ec0 10 manifest: total_size = 3918 2019-12-09T20:47:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.207+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.87 2019-12-09T20:47:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.207+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.207+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c369a8 2019-12-09T20:47:28.272 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.207+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:28.409 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.345+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.469 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.405+0000 7f63a9625ec0 10 manifest: total_size = 6042 2019-12-09T20:47:28.469 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.405+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.120 2019-12-09T20:47:28.469 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.405+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.405+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c369a8 2019-12-09T20:47:28.470 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.405+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:28.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.551+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.566+0000 7f63a9625ec0 10 manifest: total_size = 11058 2019-12-09T20:47:28.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.566+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.189 2019-12-09T20:47:28.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.566+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.566+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c369a8 2019-12-09T20:47:28.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.566+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:28.781 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.717+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.745+0000 7f63a9625ec0 10 manifest: total_size = 3987 2019-12-09T20:47:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.745+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.246 2019-12-09T20:47:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.745+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.745+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c369a8 2019-12-09T20:47:28.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.746+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:28.952 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.888+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.908+0000 7f63a9625ec0 10 manifest: total_size = 6756 2019-12-09T20:47:28.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.908+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.322 2019-12-09T20:47:28.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.908+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x55d322c369a8 s->prefetch_data=0 2019-12-09T20:47:28.973 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.908+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c369a8 2019-12-09T20:47:28.973 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:28.908+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 2019-12-09T20:47:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.048+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-conf[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1]) start _multipart_[] num_entries 1001 2019-12-09T20:47:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.048+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:29.174 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.110+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-conf[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:29.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.161+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-conf state=0x55d322c2f1f0 s->prefetch_data=0 2019-12-09T20:47:29.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.161+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-conf : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.170+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-conf info.flags=0x16 2019-12-09T20:47:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.170+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-conf to cache LRU end 2019-12-09T20:47:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.170+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.170+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.170+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-conf : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.170+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-conf from cache 2019-12-09T20:47:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.170+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=162 2019-12-09T20:47:29.243 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.179+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486850 cookie 94365309012688 notifier 4633 bl.length()=162 2019-12-09T20:47:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.831+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-conf:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.1 from cache 2019-12-09T20:47:29.895 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.831+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=219 2019-12-09T20:47:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:29.840+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486851 cookie 94365309017936 notifier 4633 bl.length()=219 2019-12-09T20:47:30.780 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.716+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-multipart-defer-complete.1 state=0x55d322c2f1f0 s->prefetch_data=0 2019-12-09T20:47:30.780 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.716+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : miss 2019-12-09T20:47:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.735+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 info.flags=0x6 2019-12-09T20:47:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.735+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 to cache LRU end 2019-12-09T20:47:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.735+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.735+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:30.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.735+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:30.800 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.735+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.745+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=229 2019-12-09T20:47:30.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.745+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 info.flags=0x1 2019-12-09T20:47:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.745+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 to cache LRU end 2019-12-09T20:47:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.745+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 state=0x55d322c36360 s->prefetch_data=0 2019-12-09T20:47:30.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.745+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 : miss 2019-12-09T20:47:30.838 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.775+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 info.flags=0x16 2019-12-09T20:47:30.838 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.775+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 to cache LRU end 2019-12-09T20:47:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.775+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.775+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.775+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.775+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:30.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.775+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.775+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:30.840 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.775+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:30.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.801+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=321 2019-12-09T20:47:30.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.801+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 info.flags=0x13 2019-12-09T20:47:30.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.801+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 to cache LRU end 2019-12-09T20:47:30.866 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.801+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:30.866 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.801+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:30.866 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.801+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 2019-12-09T20:47:30.884 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.821+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-multipart-defer-complete.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2]) start [] num_entries 1100 2019-12-09T20:47:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.834+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo[] 2019-12-09T20:47:30.897 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.834+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x55d322c36918 s->prefetch_data=0 2019-12-09T20:47:30.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.843+0000 7f63a9625ec0 10 manifest: total_size = 26079 2019-12-09T20:47:30.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.843+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.20 2019-12-09T20:47:30.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.843+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x55d322c36918 s->prefetch_data=0 2019-12-09T20:47:30.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.843+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c36918 2019-12-09T20:47:30.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.843+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 2019-12-09T20:47:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7669 rules.size()=4 2019-12-09T20:47:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=7669 2019-12-09T20:47:31.044 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=7669 stripe_ofs=7669 part_ofs=7669 rule->part_size=7671 2019-12-09T20:47:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7671 rules.size()=4 2019-12-09T20:47:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4197877 part_ofs=7669 rule->part_size=7671 2019-12-09T20:47:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=15340 stripe_ofs=15340 part_ofs=15340 rule->part_size=7672 2019-12-09T20:47:31.045 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7672 rules.size()=4 2019-12-09T20:47:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4205548 part_ofs=15340 rule->part_size=7672 2019-12-09T20:47:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=23012 stripe_ofs=23012 part_ofs=23012 rule->part_size=3067 2019-12-09T20:47:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=3067 rules.size()=4 2019-12-09T20:47:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4213220 part_ofs=23012 rule->part_size=3067 2019-12-09T20:47:31.046 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=26079 stripe_ofs=26079 part_ofs=26079 rule->part_size=3067 2019-12-09T20:47:31.047 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:30.980+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.23tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.20 2019-12-09T20:47:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.046+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-multipart-defer-complete.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2]) start _multipart_[] num_entries 1001 2019-12-09T20:47:31.109 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.046+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:31.118 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.055+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-complete.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:31.181 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.117+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-multipart-defer-complete.1 state=0x55d322c2f1f0 s->prefetch_data=0 2019-12-09T20:47:31.181 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.117+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:31.196 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.133+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 info.flags=0x16 2019-12-09T20:47:31.196 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.133+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 to cache LRU end 2019-12-09T20:47:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.133+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.133+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.133+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:31.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.133+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 from cache 2019-12-09T20:47:31.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.133+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=191 2019-12-09T20:47:31.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.184+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486848 cookie 94365309014736 notifier 4633 bl.length()=191 2019-12-09T20:47:31.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.597+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2 from cache 2019-12-09T20:47:31.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.597+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=248 2019-12-09T20:47:31.671 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.608+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486852 cookie 94365309037808 notifier 4633 bl.length()=248 2019-12-09T20:47:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.895+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-multipart-defer-update-complete.1 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:31.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.895+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : miss 2019-12-09T20:47:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.907+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 info.flags=0x6 2019-12-09T20:47:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.907+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 to cache LRU end 2019-12-09T20:47:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.907+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:31.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.907+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:31.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.907+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:31.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.907+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:31.984 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.920+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=236 2019-12-09T20:47:31.984 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.920+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 info.flags=0x1 2019-12-09T20:47:31.984 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.920+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 to cache LRU end 2019-12-09T20:47:31.984 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.920+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 state=0x55d322c2f1f0 s->prefetch_data=0 2019-12-09T20:47:31.984 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.920+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 : miss 2019-12-09T20:47:31.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.936+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 info.flags=0x16 2019-12-09T20:47:32.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.936+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 to cache LRU end 2019-12-09T20:47:32.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.936+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:32.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.936+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:32.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.936+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:32.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.936+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:32.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.936+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:32.001 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.936+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:32.001 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.936+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:32.018 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.955+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=328 2019-12-09T20:47:32.019 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.955+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 info.flags=0x13 2019-12-09T20:47:32.019 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.955+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 to cache LRU end 2019-12-09T20:47:32.019 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.955+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:32.019 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.955+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:32.019 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.955+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 2019-12-09T20:47:32.043 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:31.979+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-multipart-defer-update-complete.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3]) start [] num_entries 1100 2019-12-09T20:47:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.005+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo[] 2019-12-09T20:47:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.005+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x55d322bb9368 s->prefetch_data=0 2019-12-09T20:47:32.081 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.018+0000 7f63a9625ec0 10 manifest: total_size = 26081 2019-12-09T20:47:32.081 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.018+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.40 2019-12-09T20:47:32.081 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.018+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x55d322bb9368 s->prefetch_data=0 2019-12-09T20:47:32.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.018+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bb9368 2019-12-09T20:47:32.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.018+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 2019-12-09T20:47:32.245 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7671 rules.size()=4 2019-12-09T20:47:32.245 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=7671 2019-12-09T20:47:32.245 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=7671 stripe_ofs=7671 part_ofs=7671 rule->part_size=7669 2019-12-09T20:47:32.245 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7669 rules.size()=4 2019-12-09T20:47:32.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4197879 part_ofs=7671 rule->part_size=7669 2019-12-09T20:47:32.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=15340 stripe_ofs=15340 part_ofs=15340 rule->part_size=7674 2019-12-09T20:47:32.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7674 rules.size()=4 2019-12-09T20:47:32.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4205548 part_ofs=15340 rule->part_size=7674 2019-12-09T20:47:32.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=23014 stripe_ofs=23014 part_ofs=23014 rule->part_size=3067 2019-12-09T20:47:32.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=3067 rules.size()=4 2019-12-09T20:47:32.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4213222 part_ofs=23014 rule->part_size=3067 2019-12-09T20:47:32.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=26081 stripe_ofs=26081 part_ofs=26081 rule->part_size=3067 2019-12-09T20:47:32.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.181+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.25tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.40 2019-12-09T20:47:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.269+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-multipart-defer-update-complete.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3]) start _multipart_[] num_entries 1001 2019-12-09T20:47:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.269+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:32.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.287+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-update-complete.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:32.407 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.343+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-multipart-defer-update-complete.1 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:32.407 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.343+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:32.418 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.354+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 info.flags=0x16 2019-12-09T20:47:32.418 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.354+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 to cache LRU end 2019-12-09T20:47:32.418 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.354+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:32.418 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.354+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:32.419 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.354+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:32.419 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.354+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 from cache 2019-12-09T20:47:32.419 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.354+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=198 2019-12-09T20:47:32.431 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.368+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486853 cookie 94365309017936 notifier 4633 bl.length()=198 2019-12-09T20:47:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.749+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3 from cache 2019-12-09T20:47:32.813 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.749+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=255 2019-12-09T20:47:32.828 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.764+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486852 cookie 94365309023152 notifier 4633 bl.length()=255 2019-12-09T20:47:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.999+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-multipart-index-versioning.1 state=0x55d322c2f1f0 s->prefetch_data=0 2019-12-09T20:47:33.063 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:32.999+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : miss 2019-12-09T20:47:33.074 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.011+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 info.flags=0x6 2019-12-09T20:47:33.074 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.011+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 to cache LRU end 2019-12-09T20:47:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.011+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.011+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.011+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:33.075 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.011+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:33.086 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.023+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=231 2019-12-09T20:47:33.086 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.023+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 info.flags=0x1 2019-12-09T20:47:33.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.023+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 to cache LRU end 2019-12-09T20:47:33.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.023+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 state=0x55d322c36360 s->prefetch_data=0 2019-12-09T20:47:33.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.023+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 : miss 2019-12-09T20:47:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.035+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 info.flags=0x16 2019-12-09T20:47:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.035+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 to cache LRU end 2019-12-09T20:47:33.099 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.035+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.035+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.035+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:33.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.035+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.035+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.035+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.036+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.052+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=323 2019-12-09T20:47:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.052+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 info.flags=0x13 2019-12-09T20:47:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.052+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 to cache LRU end 2019-12-09T20:47:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.052+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:33.116 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.052+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:33.116 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.052+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.063+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-multipart-index-versioning.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4]) start [] num_entries 1100 2019-12-09T20:47:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.085+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo[35MrFFsJr9erizVYLerw.HCGjJsUsmz] 2019-12-09T20:47:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.085+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.164 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.100+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-multipart-index-versioning.1:foo state=0x55d322bb9368 s->prefetch_data=0 2019-12-09T20:47:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.113+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to awygtvey0880w49z61pl904qznbx5atw 2019-12-09T20:47:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.113+0000 7f63a9625ec0 20 get_obj_state_impl: setting s->olh_tag to oiwteledt9gtmz2pcjuznydav64hz881 2019-12-09T20:47:33.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.113+0000 7f63a9625ec0 20 bucket_index_guard_olh_op(): olh_state.olh_tag=oiwteledt9gtmz2pcjuznydav64hz881 2019-12-09T20:47:33.228 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.165+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.286 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.223+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.287 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.223+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.298 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.235+0000 7f63a9625ec0 20 olh_log_entry: epoch=3 op=2 key=[] 2019-12-09T20:47:33.298 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.235+0000 7f63a9625ec0 20 olh_log_entry: epoch=3 op=3 key=foo[35MrFFsJr9erizVYLerw.HCGjJsUsmz] 2019-12-09T20:47:33.299 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.235+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-multipart-index-versioning.1:_:35MrFFsJr9erizVYLerw.HCGjJsUsmz_foo state=0x55d322c3c198 s->prefetch_data=0 2019-12-09T20:47:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.246+0000 7f63a9625ec0 10 manifest: total_size = 886297 2019-12-09T20:47:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.246+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.55 2019-12-09T20:47:33.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.246+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-multipart-index-versioning.1:_:35MrFFsJr9erizVYLerw.HCGjJsUsmz_foo state=0x55d322c3c198 s->prefetch_data=0 2019-12-09T20:47:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.246+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3c198 2019-12-09T20:47:33.310 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.246+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.413 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2019-12-09T20:47:33.413 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=246247 2019-12-09T20:47:33.413 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2019-12-09T20:47:33.413 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2019-12-09T20:47:33.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4436455 part_ofs=246247 rule->part_size=246247 2019-12-09T20:47:33.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=246247 2019-12-09T20:47:33.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2019-12-09T20:47:33.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4682702 part_ofs=492494 rule->part_size=246247 2019-12-09T20:47:33.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=738741 stripe_ofs=738741 part_ofs=738741 rule->part_size=147556 2019-12-09T20:47:33.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=147556 rules.size()=2 2019-12-09T20:47:33.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4928949 part_ofs=738741 rule->part_size=147556 2019-12-09T20:47:33.415 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.349+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=886297 stripe_ofs=886297 part_ofs=886297 rule->part_size=147556 2019-12-09T20:47:33.415 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.350+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.5tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.55 2019-12-09T20:47:33.546 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.483+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.547 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.483+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.594+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.636+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 2019-12-09T20:47:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.636+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-multipart-index-versioning.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4]) start _multipart_[] num_entries 1001 2019-12-09T20:47:33.700 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.636+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:33.710 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.647+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-index-versioning.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:33.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.696+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-multipart-index-versioning.1 state=0x55d322c2f1f0 s->prefetch_data=0 2019-12-09T20:47:33.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.696+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:33.770 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.707+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 info.flags=0x16 2019-12-09T20:47:33.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.707+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 to cache LRU end 2019-12-09T20:47:33.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.707+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:33.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.707+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:33.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.707+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:33.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.707+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 from cache 2019-12-09T20:47:33.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.707+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=193 2019-12-09T20:47:33.782 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:33.719+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486854 cookie 94365309017936 notifier 4633 bl.length()=193 2019-12-09T20:47:34.347 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.283+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4 from cache 2019-12-09T20:47:34.347 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.283+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=250 2019-12-09T20:47:34.360 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.297+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486849 cookie 94365309042224 notifier 4633 bl.length()=250 2019-12-09T20:47:34.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.642+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-multipart-simple.1 state=0x55d322c37610 s->prefetch_data=0 2019-12-09T20:47:34.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.642+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : miss 2019-12-09T20:47:34.721 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.658+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 info.flags=0x6 2019-12-09T20:47:34.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.658+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 to cache LRU end 2019-12-09T20:47:34.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.658+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:34.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.658+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:34.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.658+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:34.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.658+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:34.737 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.674+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=221 2019-12-09T20:47:34.737 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.674+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 info.flags=0x1 2019-12-09T20:47:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.674+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 to cache LRU end 2019-12-09T20:47:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.674+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-simple.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 state=0x55d322c36360 s->prefetch_data=0 2019-12-09T20:47:34.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.674+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 : miss 2019-12-09T20:47:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.692+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 info.flags=0x16 2019-12-09T20:47:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.692+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 to cache LRU end 2019-12-09T20:47:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.692+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.692+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.692+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.692+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.692+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.692+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.692+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:34.778 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.714+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=313 2019-12-09T20:47:34.778 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.714+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 info.flags=0x13 2019-12-09T20:47:34.778 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.714+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 to cache LRU end 2019-12-09T20:47:34.779 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.714+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:34.779 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.714+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:34.779 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.714+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 2019-12-09T20:47:34.793 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.730+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-multipart-simple.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5]) start [] num_entries 1100 2019-12-09T20:47:34.813 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.750+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo[] 2019-12-09T20:47:34.814 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.750+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-multipart-simple.1:foo state=0x55d322bb9368 s->prefetch_data=0 2019-12-09T20:47:34.828 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.764+0000 7f63a9625ec0 10 manifest: total_size = 26086 2019-12-09T20:47:34.828 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.764+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.71 2019-12-09T20:47:34.828 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.764+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-multipart-simple.1:foo state=0x55d322bb9368 s->prefetch_data=0 2019-12-09T20:47:34.828 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.764+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bb9368 2019-12-09T20:47:34.828 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.764+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 2019-12-09T20:47:34.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7674 rules.size()=4 2019-12-09T20:47:34.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=7674 2019-12-09T20:47:34.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=7674 stripe_ofs=7674 part_ofs=7674 rule->part_size=7673 2019-12-09T20:47:34.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7673 rules.size()=4 2019-12-09T20:47:34.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4197882 part_ofs=7674 rule->part_size=7673 2019-12-09T20:47:34.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=15347 stripe_ofs=15347 part_ofs=15347 rule->part_size=7672 2019-12-09T20:47:34.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7672 rules.size()=4 2019-12-09T20:47:34.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4205555 part_ofs=15347 rule->part_size=7672 2019-12-09T20:47:34.991 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=23019 stripe_ofs=23019 part_ofs=23019 rule->part_size=3067 2019-12-09T20:47:34.991 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=3067 rules.size()=4 2019-12-09T20:47:34.991 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4213227 part_ofs=23019 rule->part_size=3067 2019-12-09T20:47:34.991 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=26086 stripe_ofs=26086 part_ofs=26086 rule->part_size=3067 2019-12-09T20:47:34.992 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:34.925+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.17tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.71 2019-12-09T20:47:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.039+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-multipart-simple.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5]) start _multipart_[] num_entries 1001 2019-12-09T20:47:35.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.039+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:35.117 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.054+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-simple.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:35.173 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.110+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-multipart-simple.1 state=0x55d322c36f60 s->prefetch_data=0 2019-12-09T20:47:35.174 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.110+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:35.185 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.122+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 info.flags=0x16 2019-12-09T20:47:35.185 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.122+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 to cache LRU end 2019-12-09T20:47:35.186 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.122+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:35.186 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.122+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:35.186 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.122+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:35.186 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.122+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 from cache 2019-12-09T20:47:35.186 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.122+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=183 2019-12-09T20:47:35.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.134+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486853 cookie 94365309023152 notifier 4633 bl.length()=183 2019-12-09T20:47:35.447 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.384+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5 from cache 2019-12-09T20:47:35.447 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.384+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=240 2019-12-09T20:47:35.470 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:35.407+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486851 cookie 94365309012688 notifier 4633 bl.length()=240 2019-12-09T20:47:36.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.035+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-copy.1 state=0x55d322c36f60 s->prefetch_data=0 2019-12-09T20:47:36.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.035+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : miss 2019-12-09T20:47:36.113 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.050+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 info.flags=0x6 2019-12-09T20:47:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.050+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 to cache LRU end 2019-12-09T20:47:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.050+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.050+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.050+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:36.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.050+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:36.124 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.061+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=227 2019-12-09T20:47:36.124 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.061+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 info.flags=0x1 2019-12-09T20:47:36.124 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.061+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 to cache LRU end 2019-12-09T20:47:36.125 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.061+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 state=0x55d322c37610 s->prefetch_data=0 2019-12-09T20:47:36.125 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.061+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 : miss 2019-12-09T20:47:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.076+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 info.flags=0x16 2019-12-09T20:47:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.076+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 to cache LRU end 2019-12-09T20:47:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.076+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:36.140 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.076+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.076+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.076+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.076+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.076+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:36.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.076+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:36.161 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.098+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=319 2019-12-09T20:47:36.162 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.098+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 info.flags=0x13 2019-12-09T20:47:36.162 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.098+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 to cache LRU end 2019-12-09T20:47:36.162 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.098+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:36.162 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.098+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:36.162 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.098+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:36.178 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.115+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-obj-storage-class-copy.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7]) start [] num_entries 1100 2019-12-09T20:47:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN[] 2019-12-09T20:47:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-FROZEN[] 2019-12-09T20:47:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM[] 2019-12-09T20:47:36.204 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-STANDARD[] 2019-12-09T20:47:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM[] 2019-12-09T20:47:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN[] 2019-12-09T20:47:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM[] 2019-12-09T20:47:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD[] 2019-12-09T20:47:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD[] 2019-12-09T20:47:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-FROZEN[] 2019-12-09T20:47:36.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM[] 2019-12-09T20:47:36.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-STANDARD[] 2019-12-09T20:47:36.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.140+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:36.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.162+0000 7f63a9625ec0 10 manifest: total_size = 5242880 2019-12-09T20:47:36.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.162+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.108 2019-12-09T20:47:36.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.162+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:36.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.162+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:36.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.162+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:36.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.310+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:36.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.310+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:36.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.310+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:36.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.310+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-09T20:47:36.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.310+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.16tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.108 2019-12-09T20:47:36.542 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.479+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:36.555 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.492+0000 7f63a9625ec0 10 manifest: total_size = 5242880 2019-12-09T20:47:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.492+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.109 2019-12-09T20:47:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.492+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.492+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:36.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.492+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.617+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.617+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.617+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.617+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-09T20:47:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.617+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.0tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.109 2019-12-09T20:47:36.758 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.694+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.714+0000 7f63a9625ec0 10 manifest: total_size = 5242880 2019-12-09T20:47:36.777 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.714+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _buxWHA90m7U8nrQtO660dBMQqxiXJzJ 2019-12-09T20:47:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.714+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.714+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:36.778 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.714+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:36.922 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.858+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:36.922 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.858+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:36.922 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.858+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:36.922 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.858+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-09T20:47:36.922 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.858+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.30tag is: _buxWHA90m7U8nrQtO660dBMQqxiXJzJ 2019-12-09T20:47:36.996 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.932+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.942+0000 7f63a9625ec0 10 manifest: total_size = 5242880 2019-12-09T20:47:37.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.942+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _h5WkIRQ-9R7vhngQFiIfhUIq4369nLK 2019-12-09T20:47:37.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.942+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.942+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:37.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:36.942+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:37.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.130+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:37.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.130+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-09T20:47:37.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.130+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: _h5WkIRQ-9R7vhngQFiIfhUIq4369nLK 2019-12-09T20:47:37.285 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.221+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.231+0000 7f63a9625ec0 10 manifest: total_size = 5242880 2019-12-09T20:47:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.231+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.112 2019-12-09T20:47:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.231+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.231+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.231+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.375+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.375+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:37.439 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.375+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:37.440 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.375+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-09T20:47:37.440 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.375+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.112 2019-12-09T20:47:37.488 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.424+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.439+0000 7f63a9625ec0 10 manifest: total_size = 5242880 2019-12-09T20:47:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.439+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _P5oX4wEsw7pJYePy0x5Bv7I7eef9AIh 2019-12-09T20:47:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.439+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.439+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.439+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.545+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.545+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.545+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.545+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-09T20:47:37.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.545+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.23tag is: _P5oX4wEsw7pJYePy0x5Bv7I7eef9AIh 2019-12-09T20:47:37.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.600+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.609+0000 7f63a9625ec0 10 manifest: total_size = 5242880 2019-12-09T20:47:37.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.609+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.114 2019-12-09T20:47:37.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.609+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.609+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:37.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.609+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:37.782 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.719+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.719+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.719+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.719+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-09T20:47:37.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.719+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.114 2019-12-09T20:47:37.850 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.787+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.859 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.795+0000 7f63a9625ec0 10 manifest: total_size = 5242880 2019-12-09T20:47:37.859 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.795+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _9sTH4WyWkioClwbg4ttFOMszI8Gdmix 2019-12-09T20:47:37.859 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.795+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:37.859 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.795+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:37.860 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.795+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.957+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.957+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2019-12-09T20:47:38.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:37.957+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.4tag is: _9sTH4WyWkioClwbg4ttFOMszI8Gdmix 2019-12-09T20:47:38.083 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.019+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.033+0000 7f63a9625ec0 10 manifest: total_size = 27201 2019-12-09T20:47:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.033+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.116 2019-12-09T20:47:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.033+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.033+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:38.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.033+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:38.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.148+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:38.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.162+0000 7f63a9625ec0 10 manifest: total_size = 27201 2019-12-09T20:47:38.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.162+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _zvpJd12a_GRr_JS8PwueVMRuTpQo29Q 2019-12-09T20:47:38.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.162+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:38.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.162+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:38.228 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.162+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:38.397 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.333+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.333+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=27201 stripe_ofs=27201 part_ofs=0 rule->part_size=0 2019-12-09T20:47:38.398 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.333+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.23tag is: _zvpJd12a_GRr_JS8PwueVMRuTpQo29Q 2019-12-09T20:47:38.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.436+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:38.517 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.452+0000 7f63a9625ec0 10 manifest: total_size = 27201 2019-12-09T20:47:38.517 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.452+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _CMjLq4578WSMTM1gKETnId7816mENlS 2019-12-09T20:47:38.517 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.452+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:38.518 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.452+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:38.518 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.452+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.596+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.596+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=27201 stripe_ofs=27201 part_ofs=0 rule->part_size=0 2019-12-09T20:47:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.597+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.11tag is: _CMjLq4578WSMTM1gKETnId7816mENlS 2019-12-09T20:47:38.724 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.659+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:38.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.673+0000 7f63a9625ec0 10 manifest: total_size = 27201 2019-12-09T20:47:38.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.673+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _ib1VqGkbmZUqzzGvEuAMV3UCrDIHoCW 2019-12-09T20:47:38.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.673+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x55d322bba0f8 s->prefetch_data=0 2019-12-09T20:47:38.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.673+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba0f8 2019-12-09T20:47:38.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.673+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 2019-12-09T20:47:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.791+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-obj-storage-class-copy.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7]) start _multipart_[] num_entries 1001 2019-12-09T20:47:38.856 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.791+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:38.919 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.854+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-copy.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:38.978 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.913+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-copy.1 state=0x55d322c37930 s->prefetch_data=0 2019-12-09T20:47:38.978 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.913+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:38.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.925+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 info.flags=0x16 2019-12-09T20:47:38.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.925+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 to cache LRU end 2019-12-09T20:47:38.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.925+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:38.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.925+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:38.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.925+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:38.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.925+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 from cache 2019-12-09T20:47:38.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.925+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=189 2019-12-09T20:47:38.998 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:38.933+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486855 cookie 94365309017936 notifier 4633 bl.length()=189 2019-12-09T20:47:39.367 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:39.302+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7 from cache 2019-12-09T20:47:39.368 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:39.302+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=246 2019-12-09T20:47:39.386 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:39.321+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486850 cookie 94365309042224 notifier 4633 bl.length()=246 2019-12-09T20:47:40.080 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.015+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart-copy.1 state=0x55d322c37610 s->prefetch_data=0 2019-12-09T20:47:40.080 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.015+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : miss 2019-12-09T20:47:40.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.027+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 info.flags=0x6 2019-12-09T20:47:40.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.027+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 to cache LRU end 2019-12-09T20:47:40.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.027+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:40.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.027+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:40.093 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.027+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:40.093 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.027+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:40.106 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.041+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=237 2019-12-09T20:47:40.107 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.041+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 info.flags=0x1 2019-12-09T20:47:40.107 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.041+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 to cache LRU end 2019-12-09T20:47:40.107 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.041+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:40.107 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.041+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 : miss 2019-12-09T20:47:40.122 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.057+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 info.flags=0x16 2019-12-09T20:47:40.122 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.057+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 to cache LRU end 2019-12-09T20:47:40.122 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.057+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:40.122 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.057+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.057+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.057+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.057+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.057+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:40.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.057+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.072+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=329 2019-12-09T20:47:40.137 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.072+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 info.flags=0x13 2019-12-09T20:47:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.072+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 to cache LRU end 2019-12-09T20:47:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.072+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.072+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:40.138 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.072+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:40.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.084+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-obj-storage-class-multipart-copy.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9]) start [] num_entries 1100 2019-12-09T20:47:40.183 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN[] 2019-12-09T20:47:40.183 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-FROZEN[] 2019-12-09T20:47:40.183 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM[] 2019-12-09T20:47:40.183 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN-STANDARD[] 2019-12-09T20:47:40.183 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM[] 2019-12-09T20:47:40.183 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN[] 2019-12-09T20:47:40.184 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM[] 2019-12-09T20:47:40.184 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD[] 2019-12-09T20:47:40.184 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD[] 2019-12-09T20:47:40.184 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-FROZEN[] 2019-12-09T20:47:40.184 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM[] 2019-12-09T20:47:40.185 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD-STANDARD[] 2019-12-09T20:47:40.185 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.118+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:40.193 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.128+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:40.193 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.128+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.224 2019-12-09T20:47:40.193 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.128+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:40.193 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.128+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bbaab8 2019-12-09T20:47:40.193 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.128+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:40.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:40.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:40.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:40.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:40.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:40.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:40.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:40.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:40.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-09T20:47:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-09T20:47:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.249+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.22tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.224 2019-12-09T20:47:40.401 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.336+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.345+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.345+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.225 2019-12-09T20:47:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.345+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.345+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bbaab8 2019-12-09T20:47:40.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.345+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:40.510 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:40.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:40.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:40.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-09T20:47:40.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:40.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-09T20:47:40.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.445+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.6tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.225 2019-12-09T20:47:40.558 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.493+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x55d322bba378 s->prefetch_data=0 2019-12-09T20:47:40.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.504+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:40.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.504+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _pyIq-WPxvn7M17pjhpviAE_zgoqzGME 2019-12-09T20:47:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.504+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x55d322bba378 s->prefetch_data=0 2019-12-09T20:47:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.504+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba378 2019-12-09T20:47:40.569 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.504+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:40.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.676+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.676+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.676+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.676+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-09T20:47:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.676+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:40.742 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.676+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2019-12-09T20:47:40.743 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.677+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.19tag is: _pyIq-WPxvn7M17pjhpviAE_zgoqzGME 2019-12-09T20:47:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.733+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x55d322bba378 s->prefetch_data=0 2019-12-09T20:47:40.812 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.747+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:40.812 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.747+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _U4RkBQ3RIthuZRQrGyb_UXpHjlGoMVa 2019-12-09T20:47:40.812 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.747+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x55d322bba378 s->prefetch_data=0 2019-12-09T20:47:40.813 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.747+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba378 2019-12-09T20:47:40.813 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.747+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:40.918 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.853+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.853+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-09T20:47:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.853+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.853+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2019-12-09T20:47:40.919 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.853+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.13tag is: _U4RkBQ3RIthuZRQrGyb_UXpHjlGoMVa 2019-12-09T20:47:40.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.925+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x55d322bba378 s->prefetch_data=0 2019-12-09T20:47:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.934+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.934+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.233 2019-12-09T20:47:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.934+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x55d322bba378 s->prefetch_data=0 2019-12-09T20:47:40.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.934+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bba378 2019-12-09T20:47:41.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:40.934+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:41.148 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:41.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:41.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:41.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:41.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:41.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-09T20:47:41.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:41.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-09T20:47:41.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.084+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.12tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.233 2019-12-09T20:47:41.252 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.188+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:41.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.197+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:41.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.197+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _4yuOBfWe49vaCrJ4XfdfHFb3mXMD7-1 2019-12-09T20:47:41.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.197+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:41.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.197+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bbaab8 2019-12-09T20:47:41.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.197+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.291+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:41.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.291+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.291+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.291+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-09T20:47:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.291+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.291+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2019-12-09T20:47:41.356 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.291+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.21tag is: _4yuOBfWe49vaCrJ4XfdfHFb3mXMD7-1 2019-12-09T20:47:41.422 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.358+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.372+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.373+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.235 2019-12-09T20:47:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.373+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.373+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bbaab8 2019-12-09T20:47:41.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.373+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:41.552 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:41.552 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-09T20:47:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-09T20:47:41.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.488+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.7tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.235 2019-12-09T20:47:41.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.565+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:41.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.573+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:41.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.573+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _HF6_tqeFxfrURYsNa6XPtaoONy-VdXp 2019-12-09T20:47:41.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.573+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:41.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.573+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bbaab8 2019-12-09T20:47:41.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.573+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.686+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.686+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-09T20:47:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.686+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:41.750 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.686+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2019-12-09T20:47:41.751 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.686+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.5tag is: _HF6_tqeFxfrURYsNa6XPtaoONy-VdXp 2019-12-09T20:47:41.840 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.776+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.786+0000 7f63a9625ec0 10 manifest: total_size = 16378 2019-12-09T20:47:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.786+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.242 2019-12-09T20:47:41.850 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.786+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.786+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bbaab8 2019-12-09T20:47:41.851 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.786+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:41.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7670 rules.size()=3 2019-12-09T20:47:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=7670 2019-12-09T20:47:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=7670 stripe_ofs=7670 part_ofs=7670 rule->part_size=7673 2019-12-09T20:47:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7673 rules.size()=3 2019-12-09T20:47:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4197878 part_ofs=7670 rule->part_size=7673 2019-12-09T20:47:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=15343 stripe_ofs=15343 part_ofs=15343 rule->part_size=1035 2019-12-09T20:47:41.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=1035 rules.size()=3 2019-12-09T20:47:41.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4205551 part_ofs=15343 rule->part_size=1035 2019-12-09T20:47:41.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=16378 stripe_ofs=16378 part_ofs=16378 rule->part_size=1035 2019-12-09T20:47:41.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.905+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.29tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.242 2019-12-09T20:47:42.026 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.962+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.976+0000 7f63a9625ec0 10 manifest: total_size = 16378 2019-12-09T20:47:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.976+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _naSrYAjNd7Vx6e49-Xoqgwygn64i4bC 2019-12-09T20:47:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.976+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.976+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bbaab8 2019-12-09T20:47:42.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:41.976+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.088+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.088+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=16378 stripe_ofs=16378 part_ofs=0 rule->part_size=0 2019-12-09T20:47:42.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.088+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.0tag is: _naSrYAjNd7Vx6e49-Xoqgwygn64i4bC 2019-12-09T20:47:42.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.160+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:42.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.170+0000 7f63a9625ec0 10 manifest: total_size = 16378 2019-12-09T20:47:42.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.170+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to _RqnQGDlZHxcAUG2m6on92tn58bvDsLq 2019-12-09T20:47:42.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.170+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:42.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.170+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bbaab8 2019-12-09T20:47:42.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.170+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:42.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.281+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:42.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.281+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=16378 stripe_ofs=16378 part_ofs=0 rule->part_size=0 2019-12-09T20:47:42.346 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.281+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: _RqnQGDlZHxcAUG2m6on92tn58bvDsLq 2019-12-09T20:47:42.418 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.354+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.363+0000 7f63a9625ec0 10 manifest: total_size = 16378 2019-12-09T20:47:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.363+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.245 2019-12-09T20:47:42.427 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.363+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x55d322bbaab8 s->prefetch_data=0 2019-12-09T20:47:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.363+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322bbaab8 2019-12-09T20:47:42.428 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.363+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 2019-12-09T20:47:42.518 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7670 rules.size()=3 2019-12-09T20:47:42.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=7670 2019-12-09T20:47:42.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=7670 stripe_ofs=7670 part_ofs=7670 rule->part_size=7673 2019-12-09T20:47:42.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=7673 rules.size()=3 2019-12-09T20:47:42.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4197878 part_ofs=7670 rule->part_size=7673 2019-12-09T20:47:42.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=15343 stripe_ofs=15343 part_ofs=15343 rule->part_size=1035 2019-12-09T20:47:42.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=1035 rules.size()=3 2019-12-09T20:47:42.520 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4205551 part_ofs=15343 rule->part_size=1035 2019-12-09T20:47:42.520 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=16378 stripe_ofs=16378 part_ofs=16378 rule->part_size=1035 2019-12-09T20:47:42.520 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.454+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.8tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.245 2019-12-09T20:47:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.551+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-obj-storage-class-multipart-copy.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9]) start _multipart_[] num_entries 1001 2019-12-09T20:47:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.551+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:42.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.565+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart-copy.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.649+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart-copy.1 state=0x55d322c36c30 s->prefetch_data=0 2019-12-09T20:47:42.713 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.649+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.658+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 info.flags=0x16 2019-12-09T20:47:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.658+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 to cache LRU end 2019-12-09T20:47:42.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.658+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.658+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.658+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.658+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 from cache 2019-12-09T20:47:42.723 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.658+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=199 2019-12-09T20:47:42.735 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.671+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486849 cookie 94365309019968 notifier 4633 bl.length()=199 2019-12-09T20:47:42.927 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.862+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9 from cache 2019-12-09T20:47:42.927 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.862+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.7 bl.length()=256 2019-12-09T20:47:42.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:42.874+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486851 cookie 94365309058304 notifier 4633 bl.length()=256 2019-12-09T20:47:43.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.210+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart.1 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:43.275 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.210+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : miss 2019-12-09T20:47:43.283 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.219+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 info.flags=0x6 2019-12-09T20:47:43.283 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.219+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 to cache LRU end 2019-12-09T20:47:43.283 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.219+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:43.283 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.219+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:43.283 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.219+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:43.284 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.219+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:43.294 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.230+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=232 2019-12-09T20:47:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.230+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 info.flags=0x1 2019-12-09T20:47:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.230+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 to cache LRU end 2019-12-09T20:47:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.230+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 state=0x55d322c36c30 s->prefetch_data=0 2019-12-09T20:47:43.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.230+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 : miss 2019-12-09T20:47:43.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.249+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 info.flags=0x16 2019-12-09T20:47:43.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.249+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 to cache LRU end 2019-12-09T20:47:43.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.249+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:43.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.249+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:43.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.249+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:43.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.249+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:43.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.249+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:43.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.249+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:43.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.249+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.266+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=324 2019-12-09T20:47:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.266+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 info.flags=0x13 2019-12-09T20:47:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.266+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 to cache LRU end 2019-12-09T20:47:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.266+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.266+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:43.331 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.266+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 2019-12-09T20:47:43.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.277+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-obj-storage-class-multipart.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8]) start [] num_entries 1100 2019-12-09T20:47:43.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.290+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN[] 2019-12-09T20:47:43.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.290+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM[] 2019-12-09T20:47:43.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.290+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD[] 2019-12-09T20:47:43.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.290+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:43.364 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.300+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.300+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.196 2019-12-09T20:47:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.300+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.300+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:43.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.300+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 2019-12-09T20:47:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:43.463 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:43.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-09T20:47:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-09T20:47:43.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.399+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.8tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.196 2019-12-09T20:47:43.543 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.479+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:43.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.489+0000 7f63a9625ec0 10 manifest: total_size = 11534336 2019-12-09T20:47:43.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.489+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.203 2019-12-09T20:47:43.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.489+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:43.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.489+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:43.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.489+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 2019-12-09T20:47:43.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:43.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:43.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:43.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:43.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2019-12-09T20:47:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2019-12-09T20:47:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2019-12-09T20:47:43.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2019-12-09T20:47:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2019-12-09T20:47:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2019-12-09T20:47:43.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.599+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.4tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.203 2019-12-09T20:47:43.742 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.678+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.689+0000 7f63a9625ec0 10 manifest: total_size = 45360 2019-12-09T20:47:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.689+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.210 2019-12-09T20:47:43.753 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.689+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:43.754 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.689+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:43.754 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.689+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 2019-12-09T20:47:43.840 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2019-12-09T20:47:43.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=20613 2019-12-09T20:47:43.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2019-12-09T20:47:43.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2019-12-09T20:47:43.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4210821 part_ofs=20613 rule->part_size=20613 2019-12-09T20:47:43.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=4134 2019-12-09T20:47:43.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=4134 rules.size()=2 2019-12-09T20:47:43.842 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): stripe_ofs=4231434 part_ofs=41226 rule->part_size=4134 2019-12-09T20:47:43.842 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=45360 stripe_ofs=45360 part_ofs=45360 rule->part_size=4134 2019-12-09T20:47:43.842 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.776+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.26tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.210 2019-12-09T20:47:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.832+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-obj-storage-class-multipart.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8]) start _multipart_[] num_entries 1001 2019-12-09T20:47:43.897 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.832+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:43.906 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.842+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:43.988 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.924+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart.1 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:43.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.924+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.952+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 info.flags=0x16 2019-12-09T20:47:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.952+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 to cache LRU end 2019-12-09T20:47:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.952+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:44.017 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.952+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:44.018 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.952+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:44.018 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.952+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 from cache 2019-12-09T20:47:44.018 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.952+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=194 2019-12-09T20:47:44.032 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:43.968+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486856 cookie 94365309037808 notifier 4633 bl.length()=194 2019-12-09T20:47:44.286 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.222+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8 from cache 2019-12-09T20:47:44.286 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.222+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.7 bl.length()=251 2019-12-09T20:47:44.300 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.236+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486852 cookie 94365309058304 notifier 4633 bl.length()=251 2019-12-09T20:47:44.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.538+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class.1 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:44.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.538+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : miss 2019-12-09T20:47:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.551+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 info.flags=0x6 2019-12-09T20:47:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.551+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 to cache LRU end 2019-12-09T20:47:44.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.551+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.551+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.551+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.551+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.562+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=222 2019-12-09T20:47:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.562+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 info.flags=0x1 2019-12-09T20:47:44.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.562+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 to cache LRU end 2019-12-09T20:47:44.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.562+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 state=0x55d322c368c0 s->prefetch_data=0 2019-12-09T20:47:44.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.562+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 : miss 2019-12-09T20:47:44.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.577+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 info.flags=0x16 2019-12-09T20:47:44.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.577+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 to cache LRU end 2019-12-09T20:47:44.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.577+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:44.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.577+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:44.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.577+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:44.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.577+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:44.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.577+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:44.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.577+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:44.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.577+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:44.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.597+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=314 2019-12-09T20:47:44.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.597+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 info.flags=0x13 2019-12-09T20:47:44.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.597+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 to cache LRU end 2019-12-09T20:47:44.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.597+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:44.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.597+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:44.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.597+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 2019-12-09T20:47:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.609+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-obj-storage-class.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6]) start [] num_entries 1100 2019-12-09T20:47:44.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.626+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-FROZEN[] 2019-12-09T20:47:44.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.626+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-LUKEWARM[] 2019-12-09T20:47:44.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.626+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got foo-STANDARD[] 2019-12-09T20:47:44.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.626+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:44.708 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.635+0000 7f63a9625ec0 10 manifest: total_size = 10485760 2019-12-09T20:47:44.709 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.635+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.84 2019-12-09T20:47:44.709 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.635+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:44.709 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.635+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:44.709 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.635+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 2019-12-09T20:47:44.832 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.768+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.768+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.768+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.768+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-09T20:47:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.768+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.768+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2019-12-09T20:47:44.833 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.768+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.29tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.84 2019-12-09T20:47:44.904 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.840+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.851+0000 7f63a9625ec0 10 manifest: total_size = 10485760 2019-12-09T20:47:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.851+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.85 2019-12-09T20:47:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.851+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.851+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:44.916 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.851+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 2019-12-09T20:47:45.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.956+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:45.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.956+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2019-12-09T20:47:45.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.956+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:45.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.956+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2019-12-09T20:47:45.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.956+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2019-12-09T20:47:45.022 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.956+0000 7f63a9625ec0 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2019-12-09T20:47:45.022 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:44.956+0000 7f63a9625ec0 20 garbage collection: RGWGC::send_chain - on object name: gc.13tag is: 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.85 2019-12-09T20:47:45.090 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.026+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x55d322c3c148 s->prefetch_data=0 2019-12-09T20:47:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.036+0000 7f63a9625ec0 10 manifest: total_size = 3311 2019-12-09T20:47:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.036+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.86 2019-12-09T20:47:45.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.036+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x55d322c3c148 s->prefetch_data=0 2019-12-09T20:47:45.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.036+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3c148 2019-12-09T20:47:45.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.036+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 2019-12-09T20:47:45.196 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.131+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-obj-storage-class.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6]) start _multipart_[] num_entries 1001 2019-12-09T20:47:45.196 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.131+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:45.278 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.214+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:45.351 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.287+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-obj-storage-class.1 state=0x55d322bba690 s->prefetch_data=0 2019-12-09T20:47:45.352 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.287+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.302+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 info.flags=0x16 2019-12-09T20:47:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.302+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 to cache LRU end 2019-12-09T20:47:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.302+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.302+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:45.367 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.302+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.302+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 from cache 2019-12-09T20:47:45.368 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.302+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=184 2019-12-09T20:47:45.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.320+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486857 cookie 94365309017936 notifier 4633 bl.length()=184 2019-12-09T20:47:45.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.642+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6 from cache 2019-12-09T20:47:45.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.642+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=241 2019-12-09T20:47:45.718 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:45.653+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486854 cookie 94365309023152 notifier 4633 bl.length()=241 2019-12-09T20:47:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.166+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.1 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.166+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : miss 2019-12-09T20:47:46.243 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.179+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 info.flags=0x6 2019-12-09T20:47:46.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.179+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 to cache LRU end 2019-12-09T20:47:46.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.179+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:46.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.179+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:46.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.179+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:46.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.179+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:46.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.196+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=221 2019-12-09T20:47:46.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.196+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 info.flags=0x1 2019-12-09T20:47:46.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.196+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 to cache LRU end 2019-12-09T20:47:46.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.196+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 state=0x55d322c368c0 s->prefetch_data=0 2019-12-09T20:47:46.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.196+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 : miss 2019-12-09T20:47:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.216+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 info.flags=0x16 2019-12-09T20:47:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.216+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 to cache LRU end 2019-12-09T20:47:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.216+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.216+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.216+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:46.281 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.216+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.216+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.216+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:46.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.216+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:46.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.232+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=313 2019-12-09T20:47:46.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.232+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 info.flags=0x13 2019-12-09T20:47:46.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.232+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 to cache LRU end 2019-12-09T20:47:46.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.232+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:46.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.232+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:46.298 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.232+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 2019-12-09T20:47:46.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.244+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-small-obj-data.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10]) start [] num_entries 1100 2019-12-09T20:47:46.326 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.262+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got __[] 2019-12-09T20:47:46.326 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.262+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got __ _[] 2019-12-09T20:47:46.326 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.262+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got ___[] 2019-12-09T20:47:46.326 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.262+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got obj[] 2019-12-09T20:47:46.327 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.262+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.1:__ state=0x55d322c3c148 s->prefetch_data=0 2019-12-09T20:47:46.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.277+0000 7f63a9625ec0 10 manifest: total_size = 1048576 2019-12-09T20:47:46.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.277+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.305 2019-12-09T20:47:46.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.277+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.1:__ state=0x55d322c3c148 s->prefetch_data=0 2019-12-09T20:47:46.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.277+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3c148 2019-12-09T20:47:46.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.277+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 2019-12-09T20:47:46.444 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.380+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.390+0000 7f63a9625ec0 10 manifest: total_size = 49187 2019-12-09T20:47:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.390+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.307 2019-12-09T20:47:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.390+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.390+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.391+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 2019-12-09T20:47:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.521+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.1:___ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:46.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.531+0000 7f63a9625ec0 10 manifest: total_size = 110 2019-12-09T20:47:46.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.531+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.306 2019-12-09T20:47:46.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.531+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.1:___ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:46.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.531+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:46.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.531+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 2019-12-09T20:47:46.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.683+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.1:obj state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:46.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.692+0000 7f63a9625ec0 10 manifest: total_size = 110 2019-12-09T20:47:46.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.692+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.304 2019-12-09T20:47:46.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.692+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.1:obj state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:46.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.692+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:46.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.692+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 2019-12-09T20:47:46.929 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.864+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-small-obj-data.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10]) start _multipart_[] num_entries 1001 2019-12-09T20:47:46.930 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.864+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:46.982 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.917+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.1[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:47.032 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.968+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.1 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:47.033 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.968+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.987+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 info.flags=0x16 2019-12-09T20:47:47.051 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.987+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 to cache LRU end 2019-12-09T20:47:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.987+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.987+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.987+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.987+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 from cache 2019-12-09T20:47:47.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:46.987+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=181 2019-12-09T20:47:47.067 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.003+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486849 cookie 94365309014736 notifier 4633 bl.length()=181 2019-12-09T20:47:47.431 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.367+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.10 from cache 2019-12-09T20:47:47.431 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.367+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=239 2019-12-09T20:47:47.453 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.389+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486852 cookie 94365309012688 notifier 4633 bl.length()=239 2019-12-09T20:47:47.775 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.711+0000 7f63677de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:47:47.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.765+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.2 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:47.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.765+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : miss 2019-12-09T20:47:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.784+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 info.flags=0x6 2019-12-09T20:47:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.784+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 to cache LRU end 2019-12-09T20:47:47.848 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.784+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:47.849 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.784+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:47.849 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.784+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:47.849 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.784+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.800+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=221 2019-12-09T20:47:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.800+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 info.flags=0x1 2019-12-09T20:47:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.800+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 to cache LRU end 2019-12-09T20:47:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.800+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.2:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 state=0x55d322c368c0 s->prefetch_data=0 2019-12-09T20:47:47.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.800+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 : miss 2019-12-09T20:47:47.885 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.820+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 info.flags=0x16 2019-12-09T20:47:47.885 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.820+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 to cache LRU end 2019-12-09T20:47:47.885 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.820+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:47.885 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.820+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:47.886 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.820+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:47.886 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.820+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:47.886 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.820+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:47.886 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.820+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:47.886 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.820+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:47.906 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.842+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=313 2019-12-09T20:47:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.842+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 info.flags=0x13 2019-12-09T20:47:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.842+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 to cache LRU end 2019-12-09T20:47:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.842+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.842+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:47.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.842+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 2019-12-09T20:47:47.928 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.861+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-small-obj-data.2[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11]) start [] num_entries 1100 2019-12-09T20:47:47.945 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.881+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got __[] 2019-12-09T20:47:47.945 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.881+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got __ _[] 2019-12-09T20:47:47.946 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.881+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got ___[] 2019-12-09T20:47:47.946 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.881+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got obj[] 2019-12-09T20:47:47.946 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.881+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.2:__ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.893+0000 7f63a9625ec0 10 manifest: total_size = 524288 2019-12-09T20:47:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.893+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.312 2019-12-09T20:47:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.893+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.2:__ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.893+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:47.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:47.893+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 2019-12-09T20:47:48.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.063+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.087+0000 7f63a9625ec0 10 manifest: total_size = 62 2019-12-09T20:47:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.087+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.314 2019-12-09T20:47:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.087+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.087+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.087+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 2019-12-09T20:47:48.378 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.313+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.2:___ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:48.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.325+0000 7f63a9625ec0 10 manifest: total_size = 24595 2019-12-09T20:47:48.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.325+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.313 2019-12-09T20:47:48.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.325+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.2:___ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:48.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.325+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:48.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.325+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 2019-12-09T20:47:48.559 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.495+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.2:obj state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.503+0000 7f63a9625ec0 10 manifest: total_size = 62 2019-12-09T20:47:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.503+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.311 2019-12-09T20:47:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.503+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.2:obj state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.503+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:48.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.503+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 2019-12-09T20:47:48.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.661+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-small-obj-data.2[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11]) start _multipart_[] num_entries 1001 2019-12-09T20:47:48.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.661+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:48.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.718+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.2[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.778+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.2 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:48.844 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.778+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:48.854 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.789+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 info.flags=0x16 2019-12-09T20:47:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.789+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 to cache LRU end 2019-12-09T20:47:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.789+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.789+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.789+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.789+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 from cache 2019-12-09T20:47:48.855 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.789+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=181 2019-12-09T20:47:48.869 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.804+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486850 cookie 94365309014736 notifier 4633 bl.length()=181 2019-12-09T20:47:48.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.893+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.11 from cache 2019-12-09T20:47:48.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.893+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=239 2019-12-09T20:47:48.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:48.905+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486853 cookie 94365309012688 notifier 4633 bl.length()=239 2019-12-09T20:47:49.484 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.419+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.3 state=0x55d322c368c0 s->prefetch_data=0 2019-12-09T20:47:49.485 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.419+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : miss 2019-12-09T20:47:49.504 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.438+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 info.flags=0x6 2019-12-09T20:47:49.504 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.438+0000 7f63a9625ec0 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 to cache LRU end 2019-12-09T20:47:49.504 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.438+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:49.504 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.438+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.438+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : type miss (requested=0x1, cached=0x6) 2019-12-09T20:47:49.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.438+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:49.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.453+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=221 2019-12-09T20:47:49.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.453+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 info.flags=0x1 2019-12-09T20:47:49.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.453+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 to cache LRU end 2019-12-09T20:47:49.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.453+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb35f0 obj=default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.3:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 state=0x55d322c313c0 s->prefetch_data=0 2019-12-09T20:47:49.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.453+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 : miss 2019-12-09T20:47:49.536 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.471+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 info.flags=0x16 2019-12-09T20:47:49.537 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.471+0000 7f63a9625ec0 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 to cache LRU end 2019-12-09T20:47:49.537 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.471+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:49.537 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.471+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:49.537 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.471+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:49.537 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.471+0000 7f63a9625ec0 20 Read xattr: user.rgw.acl 2019-12-09T20:47:49.537 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.471+0000 7f63a9625ec0 20 Read xattr: user.rgw.idtag 2019-12-09T20:47:49.537 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.471+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 : type miss (requested=0x13, cached=0x16) 2019-12-09T20:47:49.538 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.471+0000 7f63a9625ec0 20 rados->read ofs=0 len=0 2019-12-09T20:47:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.493+0000 7f63a9625ec0 20 rados_obj.operate() r=0 bl.length=313 2019-12-09T20:47:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.493+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 info.flags=0x13 2019-12-09T20:47:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.493+0000 7f63a9625ec0 10 moving default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 to cache LRU end 2019-12-09T20:47:49.559 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.493+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.493+0000 7f63a9625ec0 10 updating xattr: name=user.rgw.acl bl.length()=209 2019-12-09T20:47:49.560 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.493+0000 7f63a9625ec0 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 2019-12-09T20:47:49.570 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.505+0000 7f63a9625ec0 10 cls_bucket_list_unordered :test-client.0-r-test-small-obj-data.3[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12]) start [] num_entries 1100 2019-12-09T20:47:49.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.519+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got __[] 2019-12-09T20:47:49.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.519+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got __ _[] 2019-12-09T20:47:49.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.519+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got ___[] 2019-12-09T20:47:49.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.519+0000 7f63a9625ec0 10 RGWRados::cls_bucket_list_unordered: got obj[] 2019-12-09T20:47:49.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.519+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.3:__ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:49.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.530+0000 7f63a9625ec0 10 manifest: total_size = 0 2019-12-09T20:47:49.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.530+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.319 2019-12-09T20:47:49.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.530+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.3:__ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:49.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.530+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:49.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.530+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 2019-12-09T20:47:49.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.666+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:49.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.694+0000 7f63a9625ec0 10 manifest: total_size = 0 2019-12-09T20:47:49.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.695+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.321 2019-12-09T20:47:49.761 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.695+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:49.761 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.695+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:49.761 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.695+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 2019-12-09T20:47:50.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.935+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.3:___ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:50.025 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.960+0000 7f63a9625ec0 10 manifest: total_size = 0 2019-12-09T20:47:50.026 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.960+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.320 2019-12-09T20:47:50.026 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.960+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.3:___ state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:50.026 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.960+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:50.026 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:49.960+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 2019-12-09T20:47:50.202 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.137+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.3:obj state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:50.212 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.146+0000 7f63a9625ec0 10 manifest: total_size = 0 2019-12-09T20:47:50.212 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.146+0000 7f63a9625ec0 20 get_obj_state: setting s->obj_tag to 1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.318 2019-12-09T20:47:50.212 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.146+0000 7f63a9625ec0 20 get_obj_state: rctx=0x7ffed6fb3610 obj=test-client.0-r-test-small-obj-data.3:obj state=0x55d322c3b788 s->prefetch_data=0 2019-12-09T20:47:50.212 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.146+0000 7f63a9625ec0 20 prepare_atomic_modification: state is not atomic. state=0x55d322c3b788 2019-12-09T20:47:50.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.146+0000 7f63a9625ec0 20 bucket index object: default.rgw.buckets.index:.dir.1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 2019-12-09T20:47:50.362 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.296+0000 7f63a9625ec0 10 cls_bucket_list_ordered :test-client.0-r-test-small-obj-data.3[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12]) start _multipart_[] num_entries 1001 2019-12-09T20:47:50.362 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.296+0000 7f63a9625ec0 10 cls_bucket_list_ordered request from each of 1 shard(s) for 1002 entries to get 1001 total entries 2019-12-09T20:47:50.437 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.372+0000 7f63a9625ec0 20 abort_bucket_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.3[1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12])"; objs.size()=0; is_truncated=0 2019-12-09T20:47:50.486 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.420+0000 7f63a9625ec0 20 get_system_obj_state: rctx=0x7ffed6fb2fa8 obj=default.rgw.meta:root:test-client.0-r-test-small-obj-data.3 state=0x55d322c368c0 s->prefetch_data=0 2019-12-09T20:47:50.486 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.420+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : type miss (requested=0x16, cached=0x3) 2019-12-09T20:47:50.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.440+0000 7f63a9625ec0 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 info.flags=0x16 2019-12-09T20:47:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.440+0000 7f63a9625ec0 10 moving default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 to cache LRU end 2019-12-09T20:47:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.440+0000 7f63a9625ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T20:47:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.440+0000 7f63a9625ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T20:47:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.440+0000 7f63a9625ec0 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : hit (requested=0x11, cached=0x17) 2019-12-09T20:47:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.440+0000 7f63a9625ec0 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 from cache 2019-12-09T20:47:50.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.440+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=181 2019-12-09T20:47:50.522 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.457+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486851 cookie 94365309014736 notifier 4633 bl.length()=181 2019-12-09T20:47:50.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.604+0000 7f63a9625ec0 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.12 from cache 2019-12-09T20:47:50.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.604+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=239 2019-12-09T20:47:50.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:50.621+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486854 cookie 94365309012688 notifier 4633 bl.length()=239 2019-12-09T20:47:51.117 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:51.052+0000 7f63a9625ec0 10 removing key index: JBDGQWYZIKQZJAIXHECI 2019-12-09T20:47:51.118 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:51.052+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=170 2019-12-09T20:47:51.132 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:51.067+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486855 cookie 94365309023152 notifier 4633 bl.length()=170 2019-12-09T20:47:51.320 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:51.254+0000 7f63a9625ec0 10 removing email index: ragweed.client.0+test@test.test 2019-12-09T20:47:51.320 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:51.254+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=182 2019-12-09T20:47:51.337 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:51.272+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486858 cookie 94365309017936 notifier 4633 bl.length()=182 2019-12-09T20:47:51.545 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:51.480+0000 7f63a9625ec0 10 removing user buckets index 2019-12-09T20:47:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:51.480+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=173 2019-12-09T20:47:51.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:51.491+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486859 cookie 94365309037808 notifier 4633 bl.length()=173 2019-12-09T20:47:52.370 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.305+0000 7f63a9625ec0 10 removing user index: ragweed.client.0 2019-12-09T20:47:52.370 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.305+0000 7f63a9625ec0 10 removing default.rgw.meta+users.uid+ragweed.client.0 from cache 2019-12-09T20:47:52.370 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.305+0000 7f63a9625ec0 10 distributing notification oid=default.rgw.control:notify.4 bl.length()=165 2019-12-09T20:47:52.388 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.324+0000 7f6377fff700 10 RGWWatcher::handle_notify() notify_id 592705486856 cookie 94365309023152 notifier 4633 bl.length()=165 2019-12-09T20:47:52.582 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.517+0000 7f63a9625ec0 20 remove_watcher() i=0 2019-12-09T20:47:52.582 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.517+0000 7f63a9625ec0 2 removed watcher, disabling cache 2019-12-09T20:47:52.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.576+0000 7f63a9625ec0 20 remove_watcher() i=1 2019-12-09T20:47:52.692 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.627+0000 7f63a9625ec0 20 remove_watcher() i=2 2019-12-09T20:47:52.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.692+0000 7f63a9625ec0 20 remove_watcher() i=3 2019-12-09T20:47:52.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.757+0000 7f63a9625ec0 20 remove_watcher() i=4 2019-12-09T20:47:52.884 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.819+0000 7f63a9625ec0 20 remove_watcher() i=5 2019-12-09T20:47:52.943 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.878+0000 7f63a9625ec0 20 remove_watcher() i=6 2019-12-09T20:47:53.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:47:52.941+0000 7f63a9625ec0 20 remove_watcher() i=7 2019-12-09T20:47:53.016 INFO:tasks.ragweed:Removing ragweed... 2019-12-09T20:47:53.016 INFO:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/ragweed 2019-12-09T20:47:53.098 INFO:teuthology.run_tasks:Running task s3tests-java... 2019-12-09T20:47:53.112 DEBUG:tasks.s3tests_java:S3 Tests Java: __INIT__ 2019-12-09T20:47:53.113 DEBUG:tasks.s3tests_java:S3 Tests Java: SETUP 2019-12-09T20:47:53.113 INFO:tasks.s3tests_java:S3 Tests Java: Downloading test suite... 2019-12-09T20:47:53.113 INFO:teuthology.orchestra.run.smithi067:> git clone -b master https://github.com/ceph/java_s3tests.git /home/ubuntu/cephtest/s3-tests-java 2019-12-09T20:47:53.161 INFO:teuthology.orchestra.run.smithi067.stdout:Cloning into '/home/ubuntu/cephtest/s3-tests-java'... 2019-12-09T20:47:56.858 INFO:tasks.s3tests_java:S3 Tests Java: Installing required packages... 2019-12-09T20:47:56.858 INFO:teuthology.orchestra.run.smithi067:> /home/ubuntu/cephtest/s3-tests-java/bootstrap.sh 2019-12-09T20:47:59.314 INFO:teuthology.orchestra.run.smithi067.stdout: Gradle will be installed under /opt/gradle 2019-12-09T20:47:59.363 INFO:teuthology.orchestra.run.smithi067.stderr:--2019-12-09 20:47:59-- https://services.gradle.org/distributions/gradle-4.7-bin.zip 2019-12-09T20:47:59.382 INFO:teuthology.orchestra.run.smithi067.stderr:Resolving services.gradle.org (services.gradle.org)... 104.18.190.9, 104.18.191.9, 2606:4700::6812:bf09, ... 2019-12-09T20:47:59.390 INFO:teuthology.orchestra.run.smithi067.stderr:Connecting to services.gradle.org (services.gradle.org)|104.18.190.9|:443... connected. 2019-12-09T20:47:59.423 INFO:teuthology.orchestra.run.smithi067.stderr:HTTP request sent, awaiting response... 301 Moved Permanently 2019-12-09T20:47:59.423 INFO:teuthology.orchestra.run.smithi067.stderr:Location: https://downloads.gradle-dn.com/distributions/gradle-4.7-bin.zip [following] 2019-12-09T20:47:59.423 INFO:teuthology.orchestra.run.smithi067.stderr:--2019-12-09 20:47:59-- https://downloads.gradle-dn.com/distributions/gradle-4.7-bin.zip 2019-12-09T20:47:59.434 INFO:teuthology.orchestra.run.smithi067.stderr:Resolving downloads.gradle-dn.com (downloads.gradle-dn.com)... 104.17.160.20, 104.17.159.20, 2606:4700::6811:9f14, ... 2019-12-09T20:47:59.441 INFO:teuthology.orchestra.run.smithi067.stderr:Connecting to downloads.gradle-dn.com (downloads.gradle-dn.com)|104.17.160.20|:443... connected. 2019-12-09T20:47:59.479 INFO:teuthology.orchestra.run.smithi067.stderr:HTTP request sent, awaiting response... 200 OK 2019-12-09T20:47:59.480 INFO:teuthology.orchestra.run.smithi067.stderr:Length: 75323957 (72M) [application/zip] 2019-12-09T20:47:59.480 INFO:teuthology.orchestra.run.smithi067.stderr:Saving to: ‘gradle-4.7-bin.zip’ 2019-12-09T20:47:59.480 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T20:47:59.489 INFO:teuthology.orchestra.run.smithi067.stderr: 0K .......... .......... .......... .......... .......... 0% 6.39M 11s 2019-12-09T20:47:59.490 INFO:teuthology.orchestra.run.smithi067.stderr: 50K .......... .......... .......... .......... .......... 0% 141M 6s 2019-12-09T20:47:59.495 INFO:teuthology.orchestra.run.smithi067.stderr: 100K .......... .......... .......... .......... .......... 0% 7.13M 7s 2019-12-09T20:48:00.793 INFO:teuthology.orchestra.run.smithi067.stderr: 150K .......... .......... .......... .......... .......... 0% 152M 6s 2019-12-09T20:48:00.793 INFO:teuthology.orchestra.run.smithi067.stderr: 200K .......... .......... .......... .......... .......... 0% 118M 5s 2019-12-09T20:48:00.794 INFO:teuthology.orchestra.run.smithi067.stderr: 250K .......... .......... .......... .......... .......... 0% 130M 4s 2019-12-09T20:48:00.794 INFO:teuthology.orchestra.run.smithi067.stderr: 300K .......... .......... .......... .......... .......... 0% 142M 3s 2019-12-09T20:48:00.794 INFO:teuthology.orchestra.run.smithi067.stderr: 350K .......... .......... .......... .......... .......... 0% 7.66M 4s 2019-12-09T20:48:00.794 INFO:teuthology.orchestra.run.smithi067.stderr: 400K .......... .......... .......... .......... .......... 0% 122M 4s 2019-12-09T20:48:00.794 INFO:teuthology.orchestra.run.smithi067.stderr: 450K .......... .......... .......... .......... .......... 0% 157M 3s 2019-12-09T20:48:00.794 INFO:teuthology.orchestra.run.smithi067.stderr: 500K .......... .......... .......... .......... .......... 0% 116M 3s 2019-12-09T20:48:00.795 INFO:teuthology.orchestra.run.smithi067.stderr: 550K .......... .......... .......... .......... .......... 0% 129M 3s 2019-12-09T20:48:00.795 INFO:teuthology.orchestra.run.smithi067.stderr: 600K .......... .......... .......... .......... .......... 0% 109M 3s 2019-12-09T20:48:00.795 INFO:teuthology.orchestra.run.smithi067.stderr: 650K .......... .......... .......... .......... .......... 0% 141M 3s 2019-12-09T20:48:00.795 INFO:teuthology.orchestra.run.smithi067.stderr: 700K .......... .......... .......... .......... .......... 1% 147M 2s 2019-12-09T20:48:00.795 INFO:teuthology.orchestra.run.smithi067.stderr: 750K .......... .......... .......... .......... .......... 1% 109M 2s 2019-12-09T20:48:00.795 INFO:teuthology.orchestra.run.smithi067.stderr: 800K .......... .......... .......... .......... .......... 1% 139M 2s 2019-12-09T20:48:00.795 INFO:teuthology.orchestra.run.smithi067.stderr: 850K .......... .......... .......... .......... .......... 1% 12.7M 2s 2019-12-09T20:48:00.795 INFO:teuthology.orchestra.run.smithi067.stderr: 900K .......... .......... .......... .......... .......... 1% 150M 2s 2019-12-09T20:48:00.796 INFO:teuthology.orchestra.run.smithi067.stderr: 950K .......... .......... .......... .......... .......... 1% 101M 2s 2019-12-09T20:48:00.796 INFO:teuthology.orchestra.run.smithi067.stderr: 1000K .......... .......... .......... .......... .......... 1% 151M 2s 2019-12-09T20:48:00.796 INFO:teuthology.orchestra.run.smithi067.stderr: 1050K .......... .......... .......... .......... .......... 1% 129M 2s 2019-12-09T20:48:00.796 INFO:teuthology.orchestra.run.smithi067.stderr: 1100K .......... .......... .......... .......... .......... 1% 114M 2s 2019-12-09T20:48:00.796 INFO:teuthology.orchestra.run.smithi067.stderr: 1150K .......... .......... .......... .......... .......... 1% 115M 2s 2019-12-09T20:48:00.796 INFO:teuthology.orchestra.run.smithi067.stderr: 1200K .......... .......... .......... .......... .......... 1% 130M 2s 2019-12-09T20:48:00.797 INFO:teuthology.orchestra.run.smithi067.stderr: 1250K .......... .......... .......... .......... .......... 1% 134M 2s 2019-12-09T20:48:00.797 INFO:teuthology.orchestra.run.smithi067.stderr: 1300K .......... .......... .......... .......... .......... 1% 166M 2s 2019-12-09T20:48:00.797 INFO:teuthology.orchestra.run.smithi067.stderr: 1350K .......... .......... .......... .......... .......... 1% 126M 2s 2019-12-09T20:48:00.797 INFO:teuthology.orchestra.run.smithi067.stderr: 1400K .......... .......... .......... .......... .......... 1% 142M 2s 2019-12-09T20:48:00.797 INFO:teuthology.orchestra.run.smithi067.stderr: 1450K .......... .......... .......... .......... .......... 2% 170M 2s 2019-12-09T20:48:00.797 INFO:teuthology.orchestra.run.smithi067.stderr: 1500K .......... .......... .......... .......... .......... 2% 151M 2s 2019-12-09T20:48:00.797 INFO:teuthology.orchestra.run.smithi067.stderr: 1550K .......... .......... .......... .......... .......... 2% 123M 2s 2019-12-09T20:48:00.798 INFO:teuthology.orchestra.run.smithi067.stderr: 1600K .......... .......... .......... .......... .......... 2% 138M 2s 2019-12-09T20:48:00.798 INFO:teuthology.orchestra.run.smithi067.stderr: 1650K .......... .......... .......... .......... .......... 2% 165M 2s 2019-12-09T20:48:00.798 INFO:teuthology.orchestra.run.smithi067.stderr: 1700K .......... .......... .......... .......... .......... 2% 192M 1s 2019-12-09T20:48:00.798 INFO:teuthology.orchestra.run.smithi067.stderr: 1750K .......... .......... .......... .......... .......... 2% 151M 1s 2019-12-09T20:48:00.798 INFO:teuthology.orchestra.run.smithi067.stderr: 1800K .......... .......... .......... .......... .......... 2% 141M 1s 2019-12-09T20:48:00.798 INFO:teuthology.orchestra.run.smithi067.stderr: 1850K .......... .......... .......... .......... .......... 2% 163M 1s 2019-12-09T20:48:00.798 INFO:teuthology.orchestra.run.smithi067.stderr: 1900K .......... .......... .......... .......... .......... 2% 80.0M 1s 2019-12-09T20:48:00.799 INFO:teuthology.orchestra.run.smithi067.stderr: 1950K .......... .......... .......... .......... .......... 2% 103M 1s 2019-12-09T20:48:00.799 INFO:teuthology.orchestra.run.smithi067.stderr: 2000K .......... .......... .......... .......... .......... 2% 134M 1s 2019-12-09T20:48:00.799 INFO:teuthology.orchestra.run.smithi067.stderr: 2050K .......... .......... .......... .......... .......... 2% 139M 1s 2019-12-09T20:48:00.799 INFO:teuthology.orchestra.run.smithi067.stderr: 2100K .......... .......... .......... .......... .......... 2% 206M 1s 2019-12-09T20:48:00.799 INFO:teuthology.orchestra.run.smithi067.stderr: 2150K .......... .......... .......... .......... .......... 2% 222M 1s 2019-12-09T20:48:00.799 INFO:teuthology.orchestra.run.smithi067.stderr: 2200K .......... .......... .......... .......... .......... 3% 249M 1s 2019-12-09T20:48:00.799 INFO:teuthology.orchestra.run.smithi067.stderr: 2250K .......... .......... .......... .......... .......... 3% 276M 1s 2019-12-09T20:48:00.800 INFO:teuthology.orchestra.run.smithi067.stderr: 2300K .......... .......... .......... .......... .......... 3% 240M 1s 2019-12-09T20:48:00.800 INFO:teuthology.orchestra.run.smithi067.stderr: 2350K .......... .......... .......... .......... .......... 3% 223M 1s 2019-12-09T20:48:00.800 INFO:teuthology.orchestra.run.smithi067.stderr: 2400K .......... .......... .......... .......... .......... 3% 81.9M 1s 2019-12-09T20:48:00.800 INFO:teuthology.orchestra.run.smithi067.stderr: 2450K .......... .......... .......... .......... .......... 3% 125M 1s 2019-12-09T20:48:00.800 INFO:teuthology.orchestra.run.smithi067.stderr: 2500K .......... .......... .......... .......... .......... 3% 208M 1s 2019-12-09T20:48:00.800 INFO:teuthology.orchestra.run.smithi067.stderr: 2550K .......... .......... .......... .......... .......... 3% 172M 1s 2019-12-09T20:48:00.800 INFO:teuthology.orchestra.run.smithi067.stderr: 2600K .......... .......... .......... .......... .......... 3% 198M 1s 2019-12-09T20:48:00.800 INFO:teuthology.orchestra.run.smithi067.stderr: 2650K .......... .......... .......... .......... .......... 3% 216M 1s 2019-12-09T20:48:00.801 INFO:teuthology.orchestra.run.smithi067.stderr: 2700K .......... .......... .......... .......... .......... 3% 212M 1s 2019-12-09T20:48:00.801 INFO:teuthology.orchestra.run.smithi067.stderr: 2750K .......... .......... .......... .......... .......... 3% 167M 1s 2019-12-09T20:48:00.801 INFO:teuthology.orchestra.run.smithi067.stderr: 2800K .......... .......... .......... .......... .......... 3% 195M 1s 2019-12-09T20:48:00.801 INFO:teuthology.orchestra.run.smithi067.stderr: 2850K .......... .......... .......... .......... .......... 3% 176M 1s 2019-12-09T20:48:00.801 INFO:teuthology.orchestra.run.smithi067.stderr: 2900K .......... .......... .......... .......... .......... 4% 185M 1s 2019-12-09T20:48:00.801 INFO:teuthology.orchestra.run.smithi067.stderr: 2950K .......... .......... .......... .......... .......... 4% 164M 1s 2019-12-09T20:48:00.801 INFO:teuthology.orchestra.run.smithi067.stderr: 3000K .......... .......... .......... .......... .......... 4% 215M 1s 2019-12-09T20:48:00.802 INFO:teuthology.orchestra.run.smithi067.stderr: 3050K .......... .......... .......... .......... .......... 4% 231M 1s 2019-12-09T20:48:00.802 INFO:teuthology.orchestra.run.smithi067.stderr: 3100K .......... .......... .......... .......... .......... 4% 229M 1s 2019-12-09T20:48:00.802 INFO:teuthology.orchestra.run.smithi067.stderr: 3150K .......... .......... .......... .......... .......... 4% 179M 1s 2019-12-09T20:48:00.802 INFO:teuthology.orchestra.run.smithi067.stderr: 3200K .......... .......... .......... .......... .......... 4% 239M 1s 2019-12-09T20:48:00.802 INFO:teuthology.orchestra.run.smithi067.stderr: 3250K .......... .......... .......... .......... .......... 4% 222M 1s 2019-12-09T20:48:00.802 INFO:teuthology.orchestra.run.smithi067.stderr: 3300K .......... .......... .......... .......... .......... 4% 227M 1s 2019-12-09T20:48:00.802 INFO:teuthology.orchestra.run.smithi067.stderr: 3350K .......... .......... .......... .......... .......... 4% 180M 1s 2019-12-09T20:48:00.803 INFO:teuthology.orchestra.run.smithi067.stderr: 3400K .......... .......... .......... .......... .......... 4% 233M 1s 2019-12-09T20:48:00.803 INFO:teuthology.orchestra.run.smithi067.stderr: 3450K .......... .......... .......... .......... .......... 4% 224M 1s 2019-12-09T20:48:00.803 INFO:teuthology.orchestra.run.smithi067.stderr: 3500K .......... .......... .......... .......... .......... 4% 223M 1s 2019-12-09T20:48:00.803 INFO:teuthology.orchestra.run.smithi067.stderr: 3550K .......... .......... .......... .......... .......... 4% 185M 1s 2019-12-09T20:48:00.803 INFO:teuthology.orchestra.run.smithi067.stderr: 3600K .......... .......... .......... .......... .......... 4% 220M 1s 2019-12-09T20:48:00.803 INFO:teuthology.orchestra.run.smithi067.stderr: 3650K .......... .......... .......... .......... .......... 5% 229M 1s 2019-12-09T20:48:00.803 INFO:teuthology.orchestra.run.smithi067.stderr: 3700K .......... .......... .......... .......... .......... 5% 230M 1s 2019-12-09T20:48:00.804 INFO:teuthology.orchestra.run.smithi067.stderr: 3750K .......... .......... .......... .......... .......... 5% 205M 1s 2019-12-09T20:48:00.804 INFO:teuthology.orchestra.run.smithi067.stderr: 3800K .......... .......... .......... .......... .......... 5% 222M 1s 2019-12-09T20:48:00.804 INFO:teuthology.orchestra.run.smithi067.stderr: 3850K .......... .......... .......... .......... .......... 5% 218M 1s 2019-12-09T20:48:00.804 INFO:teuthology.orchestra.run.smithi067.stderr: 3900K .......... .......... .......... .......... .......... 5% 232M 1s 2019-12-09T20:48:00.804 INFO:teuthology.orchestra.run.smithi067.stderr: 3950K .......... .......... .......... .......... .......... 5% 199M 1s 2019-12-09T20:48:00.804 INFO:teuthology.orchestra.run.smithi067.stderr: 4000K .......... .......... .......... .......... .......... 5% 218M 1s 2019-12-09T20:48:00.804 INFO:teuthology.orchestra.run.smithi067.stderr: 4050K .......... .......... .......... .......... .......... 5% 236M 1s 2019-12-09T20:48:00.805 INFO:teuthology.orchestra.run.smithi067.stderr: 4100K .......... .......... .......... .......... .......... 5% 224M 1s 2019-12-09T20:48:00.805 INFO:teuthology.orchestra.run.smithi067.stderr: 4150K .......... .......... .......... .......... .......... 5% 176M 1s 2019-12-09T20:48:00.805 INFO:teuthology.orchestra.run.smithi067.stderr: 4200K .......... .......... .......... .......... .......... 5% 200M 1s 2019-12-09T20:48:00.805 INFO:teuthology.orchestra.run.smithi067.stderr: 4250K .......... .......... .......... .......... .......... 5% 187M 1s 2019-12-09T20:48:00.805 INFO:teuthology.orchestra.run.smithi067.stderr: 4300K .......... .......... .......... .......... .......... 5% 198M 1s 2019-12-09T20:48:00.805 INFO:teuthology.orchestra.run.smithi067.stderr: 4350K .......... .......... .......... .......... .......... 5% 170M 1s 2019-12-09T20:48:00.805 INFO:teuthology.orchestra.run.smithi067.stderr: 4400K .......... .......... .......... .......... .......... 6% 213M 1s 2019-12-09T20:48:00.805 INFO:teuthology.orchestra.run.smithi067.stderr: 4450K .......... .......... .......... .......... .......... 6% 219M 1s 2019-12-09T20:48:00.806 INFO:teuthology.orchestra.run.smithi067.stderr: 4500K .......... .......... .......... .......... .......... 6% 246M 1s 2019-12-09T20:48:00.806 INFO:teuthology.orchestra.run.smithi067.stderr: 4550K .......... .......... .......... .......... .......... 6% 196M 1s 2019-12-09T20:48:00.806 INFO:teuthology.orchestra.run.smithi067.stderr: 4600K .......... .......... .......... .......... .......... 6% 243M 1s 2019-12-09T20:48:00.806 INFO:teuthology.orchestra.run.smithi067.stderr: 4650K .......... .......... .......... .......... .......... 6% 215M 1s 2019-12-09T20:48:00.806 INFO:teuthology.orchestra.run.smithi067.stderr: 4700K .......... .......... .......... .......... .......... 6% 222M 1s 2019-12-09T20:48:00.806 INFO:teuthology.orchestra.run.smithi067.stderr: 4750K .......... .......... .......... .......... .......... 6% 183M 1s 2019-12-09T20:48:00.806 INFO:teuthology.orchestra.run.smithi067.stderr: 4800K .......... .......... .......... .......... .......... 6% 224M 1s 2019-12-09T20:48:00.807 INFO:teuthology.orchestra.run.smithi067.stderr: 4850K .......... .......... .......... .......... .......... 6% 195M 1s 2019-12-09T20:48:00.807 INFO:teuthology.orchestra.run.smithi067.stderr: 4900K .......... .......... .......... .......... .......... 6% 229M 1s 2019-12-09T20:48:00.807 INFO:teuthology.orchestra.run.smithi067.stderr: 4950K .......... .......... .......... .......... .......... 6% 194M 1s 2019-12-09T20:48:00.807 INFO:teuthology.orchestra.run.smithi067.stderr: 5000K .......... .......... .......... .......... .......... 6% 220M 1s 2019-12-09T20:48:00.807 INFO:teuthology.orchestra.run.smithi067.stderr: 5050K .......... .......... .......... .......... .......... 6% 203M 1s 2019-12-09T20:48:00.807 INFO:teuthology.orchestra.run.smithi067.stderr: 5100K .......... .......... .......... .......... .......... 7% 200M 1s 2019-12-09T20:48:00.807 INFO:teuthology.orchestra.run.smithi067.stderr: 5150K .......... .......... .......... .......... .......... 7% 182M 1s 2019-12-09T20:48:00.808 INFO:teuthology.orchestra.run.smithi067.stderr: 5200K .......... .......... .......... .......... .......... 7% 198M 1s 2019-12-09T20:48:00.808 INFO:teuthology.orchestra.run.smithi067.stderr: 5250K .......... .......... .......... .......... .......... 7% 192M 1s 2019-12-09T20:48:00.808 INFO:teuthology.orchestra.run.smithi067.stderr: 5300K .......... .......... .......... .......... .......... 7% 238M 1s 2019-12-09T20:48:00.808 INFO:teuthology.orchestra.run.smithi067.stderr: 5350K .......... .......... .......... .......... .......... 7% 190M 1s 2019-12-09T20:48:00.808 INFO:teuthology.orchestra.run.smithi067.stderr: 5400K .......... .......... .......... .......... .......... 7% 238M 1s 2019-12-09T20:48:00.808 INFO:teuthology.orchestra.run.smithi067.stderr: 5450K .......... .......... .......... .......... .......... 7% 235M 1s 2019-12-09T20:48:00.808 INFO:teuthology.orchestra.run.smithi067.stderr: 5500K .......... .......... .......... .......... .......... 7% 231M 1s 2019-12-09T20:48:00.809 INFO:teuthology.orchestra.run.smithi067.stderr: 5550K .......... .......... .......... .......... .......... 7% 198M 1s 2019-12-09T20:48:00.809 INFO:teuthology.orchestra.run.smithi067.stderr: 5600K .......... .......... .......... .......... .......... 7% 248M 1s 2019-12-09T20:48:00.809 INFO:teuthology.orchestra.run.smithi067.stderr: 5650K .......... .......... .......... .......... .......... 7% 234M 1s 2019-12-09T20:48:00.809 INFO:teuthology.orchestra.run.smithi067.stderr: 5700K .......... .......... .......... .......... .......... 7% 234M 1s 2019-12-09T20:48:00.809 INFO:teuthology.orchestra.run.smithi067.stderr: 5750K .......... .......... .......... .......... .......... 7% 204M 1s 2019-12-09T20:48:00.809 INFO:teuthology.orchestra.run.smithi067.stderr: 5800K .......... .......... .......... .......... .......... 7% 243M 1s 2019-12-09T20:48:00.809 INFO:teuthology.orchestra.run.smithi067.stderr: 5850K .......... .......... .......... .......... .......... 8% 213M 1s 2019-12-09T20:48:00.810 INFO:teuthology.orchestra.run.smithi067.stderr: 5900K .......... .......... .......... .......... .......... 8% 193M 1s 2019-12-09T20:48:00.810 INFO:teuthology.orchestra.run.smithi067.stderr: 5950K .......... .......... .......... .......... .......... 8% 170M 1s 2019-12-09T20:48:00.810 INFO:teuthology.orchestra.run.smithi067.stderr: 6000K .......... .......... .......... .......... .......... 8% 207M 1s 2019-12-09T20:48:00.810 INFO:teuthology.orchestra.run.smithi067.stderr: 6050K .......... .......... .......... .......... .......... 8% 199M 1s 2019-12-09T20:48:00.810 INFO:teuthology.orchestra.run.smithi067.stderr: 6100K .......... .......... .......... .......... .......... 8% 214M 1s 2019-12-09T20:48:00.810 INFO:teuthology.orchestra.run.smithi067.stderr: 6150K .......... .......... .......... .......... .......... 8% 186M 1s 2019-12-09T20:48:00.810 INFO:teuthology.orchestra.run.smithi067.stderr: 6200K .......... .......... .......... .......... .......... 8% 216M 1s 2019-12-09T20:48:00.811 INFO:teuthology.orchestra.run.smithi067.stderr: 6250K .......... .......... .......... .......... .......... 8% 230M 1s 2019-12-09T20:48:00.811 INFO:teuthology.orchestra.run.smithi067.stderr: 6300K .......... .......... .......... .......... .......... 8% 245M 1s 2019-12-09T20:48:00.811 INFO:teuthology.orchestra.run.smithi067.stderr: 6350K .......... .......... .......... .......... .......... 8% 169M 1s 2019-12-09T20:48:00.811 INFO:teuthology.orchestra.run.smithi067.stderr: 6400K .......... .......... .......... .......... .......... 8% 217M 1s 2019-12-09T20:48:00.811 INFO:teuthology.orchestra.run.smithi067.stderr: 6450K .......... .......... .......... .......... .......... 8% 248M 1s 2019-12-09T20:48:00.811 INFO:teuthology.orchestra.run.smithi067.stderr: 6500K .......... .......... .......... .......... .......... 8% 253M 1s 2019-12-09T20:48:00.811 INFO:teuthology.orchestra.run.smithi067.stderr: 6550K .......... .......... .......... .......... .......... 8% 214M 1s 2019-12-09T20:48:00.811 INFO:teuthology.orchestra.run.smithi067.stderr: 6600K .......... .......... .......... .......... .......... 9% 219M 1s 2019-12-09T20:48:00.812 INFO:teuthology.orchestra.run.smithi067.stderr: 6650K .......... .......... .......... .......... .......... 9% 231M 1s 2019-12-09T20:48:00.812 INFO:teuthology.orchestra.run.smithi067.stderr: 6700K .......... .......... .......... .......... .......... 9% 231M 1s 2019-12-09T20:48:00.812 INFO:teuthology.orchestra.run.smithi067.stderr: 6750K .......... .......... .......... .......... .......... 9% 230M 1s 2019-12-09T20:48:00.812 INFO:teuthology.orchestra.run.smithi067.stderr: 6800K .......... .......... .......... .......... .......... 9% 243M 1s 2019-12-09T20:48:00.812 INFO:teuthology.orchestra.run.smithi067.stderr: 6850K .......... .......... .......... .......... .......... 9% 228M 1s 2019-12-09T20:48:00.812 INFO:teuthology.orchestra.run.smithi067.stderr: 6900K .......... .......... .......... .......... .......... 9% 242M 1s 2019-12-09T20:48:00.812 INFO:teuthology.orchestra.run.smithi067.stderr: 6950K .......... .......... .......... .......... .......... 9% 216M 1s 2019-12-09T20:48:00.813 INFO:teuthology.orchestra.run.smithi067.stderr: 7000K .......... .......... .......... .......... .......... 9% 224M 1s 2019-12-09T20:48:00.813 INFO:teuthology.orchestra.run.smithi067.stderr: 7050K .......... .......... .......... .......... .......... 9% 215M 1s 2019-12-09T20:48:00.813 INFO:teuthology.orchestra.run.smithi067.stderr: 7100K .......... .......... .......... .......... .......... 9% 238M 1s 2019-12-09T20:48:00.813 INFO:teuthology.orchestra.run.smithi067.stderr: 7150K .......... .......... .......... .......... .......... 9% 217M 1s 2019-12-09T20:48:00.813 INFO:teuthology.orchestra.run.smithi067.stderr: 7200K .......... .......... .......... .......... .......... 9% 227M 1s 2019-12-09T20:48:00.813 INFO:teuthology.orchestra.run.smithi067.stderr: 7250K .......... .......... .......... .......... .......... 9% 250M 1s 2019-12-09T20:48:00.813 INFO:teuthology.orchestra.run.smithi067.stderr: 7300K .......... .......... .......... .......... .......... 9% 218M 1s 2019-12-09T20:48:00.814 INFO:teuthology.orchestra.run.smithi067.stderr: 7350K .......... .......... .......... .......... .......... 10% 180M 1s 2019-12-09T20:48:00.814 INFO:teuthology.orchestra.run.smithi067.stderr: 7400K .......... .......... .......... .......... .......... 10% 220M 1s 2019-12-09T20:48:00.814 INFO:teuthology.orchestra.run.smithi067.stderr: 7450K .......... .......... .......... .......... .......... 10% 250M 1s 2019-12-09T20:48:00.814 INFO:teuthology.orchestra.run.smithi067.stderr: 7500K .......... .......... .......... .......... .......... 10% 191M 1s 2019-12-09T20:48:00.814 INFO:teuthology.orchestra.run.smithi067.stderr: 7550K .......... .......... .......... .......... .......... 10% 186M 1s 2019-12-09T20:48:00.814 INFO:teuthology.orchestra.run.smithi067.stderr: 7600K .......... .......... .......... .......... .......... 10% 247M 1s 2019-12-09T20:48:00.814 INFO:teuthology.orchestra.run.smithi067.stderr: 7650K .......... .......... .......... .......... .......... 10% 250M 1s 2019-12-09T20:48:00.815 INFO:teuthology.orchestra.run.smithi067.stderr: 7700K .......... .......... .......... .......... .......... 10% 243M 1s 2019-12-09T20:48:00.815 INFO:teuthology.orchestra.run.smithi067.stderr: 7750K .......... .......... .......... .......... .......... 10% 184M 1s 2019-12-09T20:48:00.815 INFO:teuthology.orchestra.run.smithi067.stderr: 7800K .......... .......... .......... .......... .......... 10% 237M 1s 2019-12-09T20:48:00.815 INFO:teuthology.orchestra.run.smithi067.stderr: 7850K .......... .......... .......... .......... .......... 10% 230M 1s 2019-12-09T20:48:00.815 INFO:teuthology.orchestra.run.smithi067.stderr: 7900K .......... .......... .......... .......... .......... 10% 231M 1s 2019-12-09T20:48:00.815 INFO:teuthology.orchestra.run.smithi067.stderr: 7950K .......... .......... .......... .......... .......... 10% 208M 1s 2019-12-09T20:48:00.815 INFO:teuthology.orchestra.run.smithi067.stderr: 8000K .......... .......... .......... .......... .......... 10% 240M 1s 2019-12-09T20:48:00.816 INFO:teuthology.orchestra.run.smithi067.stderr: 8050K .......... .......... .......... .......... .......... 11% 250M 1s 2019-12-09T20:48:00.816 INFO:teuthology.orchestra.run.smithi067.stderr: 8100K .......... .......... .......... .......... .......... 11% 244M 1s 2019-12-09T20:48:00.816 INFO:teuthology.orchestra.run.smithi067.stderr: 8150K .......... .......... .......... .......... .......... 11% 225M 1s 2019-12-09T20:48:00.816 INFO:teuthology.orchestra.run.smithi067.stderr: 8200K .......... .......... .......... .......... .......... 11% 246M 1s 2019-12-09T20:48:00.816 INFO:teuthology.orchestra.run.smithi067.stderr: 8250K .......... .......... .......... .......... .......... 11% 256M 1s 2019-12-09T20:48:00.816 INFO:teuthology.orchestra.run.smithi067.stderr: 8300K .......... .......... .......... .......... .......... 11% 247M 1s 2019-12-09T20:48:00.816 INFO:teuthology.orchestra.run.smithi067.stderr: 8350K .......... .......... .......... .......... .......... 11% 212M 1s 2019-12-09T20:48:00.817 INFO:teuthology.orchestra.run.smithi067.stderr: 8400K .......... .......... .......... .......... .......... 11% 252M 1s 2019-12-09T20:48:00.817 INFO:teuthology.orchestra.run.smithi067.stderr: 8450K .......... .......... .......... .......... .......... 11% 235M 1s 2019-12-09T20:48:00.817 INFO:teuthology.orchestra.run.smithi067.stderr: 8500K .......... .......... .......... .......... .......... 11% 256M 1s 2019-12-09T20:48:00.817 INFO:teuthology.orchestra.run.smithi067.stderr: 8550K .......... .......... .......... .......... .......... 11% 213M 1s 2019-12-09T20:48:00.817 INFO:teuthology.orchestra.run.smithi067.stderr: 8600K .......... .......... .......... .......... .......... 11% 242M 1s 2019-12-09T20:48:00.817 INFO:teuthology.orchestra.run.smithi067.stderr: 8650K .......... .......... .......... .......... .......... 11% 254M 1s 2019-12-09T20:48:00.817 INFO:teuthology.orchestra.run.smithi067.stderr: 8700K .......... .......... .......... .......... .......... 11% 231M 1s 2019-12-09T20:48:00.818 INFO:teuthology.orchestra.run.smithi067.stderr: 8750K .......... .......... .......... .......... .......... 11% 211M 1s 2019-12-09T20:48:00.818 INFO:teuthology.orchestra.run.smithi067.stderr: 8800K .......... .......... .......... .......... .......... 12% 247M 1s 2019-12-09T20:48:00.818 INFO:teuthology.orchestra.run.smithi067.stderr: 8850K .......... .......... .......... .......... .......... 12% 259M 1s 2019-12-09T20:48:00.818 INFO:teuthology.orchestra.run.smithi067.stderr: 8900K .......... .......... .......... .......... .......... 12% 249M 1s 2019-12-09T20:48:00.818 INFO:teuthology.orchestra.run.smithi067.stderr: 8950K .......... .......... .......... .......... .......... 12% 170M 1s 2019-12-09T20:48:00.818 INFO:teuthology.orchestra.run.smithi067.stderr: 9000K .......... .......... .......... .......... .......... 12% 206M 1s 2019-12-09T20:48:00.818 INFO:teuthology.orchestra.run.smithi067.stderr: 9050K .......... .......... .......... .......... .......... 12% 258M 1s 2019-12-09T20:48:00.819 INFO:teuthology.orchestra.run.smithi067.stderr: 9100K .......... .......... .......... .......... .......... 12% 256M 1s 2019-12-09T20:48:00.819 INFO:teuthology.orchestra.run.smithi067.stderr: 9150K .......... .......... .......... .......... .......... 12% 185M 0s 2019-12-09T20:48:00.819 INFO:teuthology.orchestra.run.smithi067.stderr: 9200K .......... .......... .......... .......... .......... 12% 215M 0s 2019-12-09T20:48:00.819 INFO:teuthology.orchestra.run.smithi067.stderr: 9250K .......... .......... .......... .......... .......... 12% 233M 0s 2019-12-09T20:48:00.819 INFO:teuthology.orchestra.run.smithi067.stderr: 9300K .......... .......... .......... .......... .......... 12% 242M 0s 2019-12-09T20:48:00.819 INFO:teuthology.orchestra.run.smithi067.stderr: 9350K .......... .......... .......... .......... .......... 12% 217M 0s 2019-12-09T20:48:00.819 INFO:teuthology.orchestra.run.smithi067.stderr: 9400K .......... .......... .......... .......... .......... 12% 224M 0s 2019-12-09T20:48:00.819 INFO:teuthology.orchestra.run.smithi067.stderr: 9450K .......... .......... .......... .......... .......... 12% 261M 0s 2019-12-09T20:48:00.820 INFO:teuthology.orchestra.run.smithi067.stderr: 9500K .......... .......... .......... .......... .......... 12% 269M 0s 2019-12-09T20:48:00.820 INFO:teuthology.orchestra.run.smithi067.stderr: 9550K .......... .......... .......... .......... .......... 13% 210M 0s 2019-12-09T20:48:00.820 INFO:teuthology.orchestra.run.smithi067.stderr: 9600K .......... .......... .......... .......... .......... 13% 262M 0s 2019-12-09T20:48:00.820 INFO:teuthology.orchestra.run.smithi067.stderr: 9650K .......... .......... .......... .......... .......... 13% 252M 0s 2019-12-09T20:48:00.820 INFO:teuthology.orchestra.run.smithi067.stderr: 9700K .......... .......... .......... .......... .......... 13% 230M 0s 2019-12-09T20:48:00.820 INFO:teuthology.orchestra.run.smithi067.stderr: 9750K .......... .......... .......... .......... .......... 13% 205M 0s 2019-12-09T20:48:00.820 INFO:teuthology.orchestra.run.smithi067.stderr: 9800K .......... .......... .......... .......... .......... 13% 248M 0s 2019-12-09T20:48:00.821 INFO:teuthology.orchestra.run.smithi067.stderr: 9850K .......... .......... .......... .......... .......... 13% 219M 0s 2019-12-09T20:48:00.821 INFO:teuthology.orchestra.run.smithi067.stderr: 9900K .......... .......... .......... .......... .......... 13% 234M 0s 2019-12-09T20:48:00.821 INFO:teuthology.orchestra.run.smithi067.stderr: 9950K .......... .......... .......... .......... .......... 13% 210M 0s 2019-12-09T20:48:00.821 INFO:teuthology.orchestra.run.smithi067.stderr: 10000K .......... .......... .......... .......... .......... 13% 237M 0s 2019-12-09T20:48:00.821 INFO:teuthology.orchestra.run.smithi067.stderr: 10050K .......... .......... .......... .......... .......... 13% 221M 0s 2019-12-09T20:48:00.821 INFO:teuthology.orchestra.run.smithi067.stderr: 10100K .......... .......... .......... .......... .......... 13% 231M 0s 2019-12-09T20:48:00.821 INFO:teuthology.orchestra.run.smithi067.stderr: 10150K .......... .......... .......... .......... .......... 13% 188M 0s 2019-12-09T20:48:00.822 INFO:teuthology.orchestra.run.smithi067.stderr: 10200K .......... .......... .......... .......... .......... 13% 213M 0s 2019-12-09T20:48:00.822 INFO:teuthology.orchestra.run.smithi067.stderr: 10250K .......... .......... .......... .......... .......... 14% 244M 0s 2019-12-09T20:48:00.822 INFO:teuthology.orchestra.run.smithi067.stderr: 10300K .......... .......... .......... .......... .......... 14% 260M 0s 2019-12-09T20:48:00.822 INFO:teuthology.orchestra.run.smithi067.stderr: 10350K .......... .......... .......... .......... .......... 14% 194M 0s 2019-12-09T20:48:00.822 INFO:teuthology.orchestra.run.smithi067.stderr: 10400K .......... .......... .......... .......... .......... 14% 218M 0s 2019-12-09T20:48:00.822 INFO:teuthology.orchestra.run.smithi067.stderr: 10450K .......... .......... .......... .......... .......... 14% 242M 0s 2019-12-09T20:48:00.822 INFO:teuthology.orchestra.run.smithi067.stderr: 10500K .......... .......... .......... .......... .......... 14% 257M 0s 2019-12-09T20:48:00.823 INFO:teuthology.orchestra.run.smithi067.stderr: 10550K .......... .......... .......... .......... .......... 14% 221M 0s 2019-12-09T20:48:00.823 INFO:teuthology.orchestra.run.smithi067.stderr: 10600K .......... .......... .......... .......... .......... 14% 236M 0s 2019-12-09T20:48:00.823 INFO:teuthology.orchestra.run.smithi067.stderr: 10650K .......... .......... .......... .......... .......... 14% 249M 0s 2019-12-09T20:48:00.823 INFO:teuthology.orchestra.run.smithi067.stderr: 10700K .......... .......... .......... .......... .......... 14% 257M 0s 2019-12-09T20:48:00.823 INFO:teuthology.orchestra.run.smithi067.stderr: 10750K .......... .......... .......... .......... .......... 14% 216M 0s 2019-12-09T20:48:00.823 INFO:teuthology.orchestra.run.smithi067.stderr: 10800K .......... .......... .......... .......... .......... 14% 255M 0s 2019-12-09T20:48:00.823 INFO:teuthology.orchestra.run.smithi067.stderr: 10850K .......... .......... .......... .......... .......... 14% 269M 0s 2019-12-09T20:48:00.823 INFO:teuthology.orchestra.run.smithi067.stderr: 10900K .......... .......... .......... .......... .......... 14% 258M 0s 2019-12-09T20:48:00.824 INFO:teuthology.orchestra.run.smithi067.stderr: 10950K .......... .......... .......... .......... .......... 14% 221M 0s 2019-12-09T20:48:00.824 INFO:teuthology.orchestra.run.smithi067.stderr: 11000K .......... .......... .......... .......... .......... 15% 244M 0s 2019-12-09T20:48:00.824 INFO:teuthology.orchestra.run.smithi067.stderr: 11050K .......... .......... .......... .......... .......... 15% 235M 0s 2019-12-09T20:48:00.824 INFO:teuthology.orchestra.run.smithi067.stderr: 11100K .......... .......... .......... .......... .......... 15% 241M 0s 2019-12-09T20:48:00.824 INFO:teuthology.orchestra.run.smithi067.stderr: 11150K .......... .......... .......... .......... .......... 15% 202M 0s 2019-12-09T20:48:00.824 INFO:teuthology.orchestra.run.smithi067.stderr: 11200K .......... .......... .......... .......... .......... 15% 214M 0s 2019-12-09T20:48:00.824 INFO:teuthology.orchestra.run.smithi067.stderr: 11250K .......... .......... .......... .......... .......... 15% 217M 0s 2019-12-09T20:48:00.825 INFO:teuthology.orchestra.run.smithi067.stderr: 11300K .......... .......... .......... .......... .......... 15% 233M 0s 2019-12-09T20:48:00.825 INFO:teuthology.orchestra.run.smithi067.stderr: 11350K .......... .......... .......... .......... .......... 15% 218M 0s 2019-12-09T20:48:00.825 INFO:teuthology.orchestra.run.smithi067.stderr: 11400K .......... .......... .......... .......... .......... 15% 245M 0s 2019-12-09T20:48:00.825 INFO:teuthology.orchestra.run.smithi067.stderr: 11450K .......... .......... .......... .......... .......... 15% 241M 0s 2019-12-09T20:48:00.825 INFO:teuthology.orchestra.run.smithi067.stderr: 11500K .......... .......... .......... .......... .......... 15% 254M 0s 2019-12-09T20:48:00.825 INFO:teuthology.orchestra.run.smithi067.stderr: 11550K .......... .......... .......... .......... .......... 15% 222M 0s 2019-12-09T20:48:00.825 INFO:teuthology.orchestra.run.smithi067.stderr: 11600K .......... .......... .......... .......... .......... 15% 271M 0s 2019-12-09T20:48:00.826 INFO:teuthology.orchestra.run.smithi067.stderr: 11650K .......... .......... .......... .......... .......... 15% 285M 0s 2019-12-09T20:48:00.826 INFO:teuthology.orchestra.run.smithi067.stderr: 11700K .......... .......... .......... .......... .......... 15% 270M 0s 2019-12-09T20:48:00.826 INFO:teuthology.orchestra.run.smithi067.stderr: 11750K .......... .......... .......... .......... .......... 16% 235M 0s 2019-12-09T20:48:00.826 INFO:teuthology.orchestra.run.smithi067.stderr: 11800K .......... .......... .......... .......... .......... 16% 266M 0s 2019-12-09T20:48:00.826 INFO:teuthology.orchestra.run.smithi067.stderr: 11850K .......... .......... .......... .......... .......... 16% 263M 0s 2019-12-09T20:48:00.826 INFO:teuthology.orchestra.run.smithi067.stderr: 11900K .......... .......... .......... .......... .......... 16% 247M 0s 2019-12-09T20:48:00.826 INFO:teuthology.orchestra.run.smithi067.stderr: 11950K .......... .......... .......... .......... .......... 16% 223M 0s 2019-12-09T20:48:00.827 INFO:teuthology.orchestra.run.smithi067.stderr: 12000K .......... .......... .......... .......... .......... 16% 284M 0s 2019-12-09T20:48:00.827 INFO:teuthology.orchestra.run.smithi067.stderr: 12050K .......... .......... .......... .......... .......... 16% 229M 0s 2019-12-09T20:48:00.827 INFO:teuthology.orchestra.run.smithi067.stderr: 12100K .......... .......... .......... .......... .......... 16% 258M 0s 2019-12-09T20:48:00.827 INFO:teuthology.orchestra.run.smithi067.stderr: 12150K .......... .......... .......... .......... .......... 16% 240M 0s 2019-12-09T20:48:00.827 INFO:teuthology.orchestra.run.smithi067.stderr: 12200K .......... .......... .......... .......... .......... 16% 284M 0s 2019-12-09T20:48:00.827 INFO:teuthology.orchestra.run.smithi067.stderr: 12250K .......... .......... .......... .......... .......... 16% 266M 0s 2019-12-09T20:48:00.827 INFO:teuthology.orchestra.run.smithi067.stderr: 12300K .......... .......... .......... .......... .......... 16% 256M 0s 2019-12-09T20:48:00.827 INFO:teuthology.orchestra.run.smithi067.stderr: 12350K .......... .......... .......... .......... .......... 16% 245M 0s 2019-12-09T20:48:00.828 INFO:teuthology.orchestra.run.smithi067.stderr: 12400K .......... .......... .......... .......... .......... 16% 286M 0s 2019-12-09T20:48:00.828 INFO:teuthology.orchestra.run.smithi067.stderr: 12450K .......... .......... .......... .......... .......... 16% 272M 0s 2019-12-09T20:48:00.828 INFO:teuthology.orchestra.run.smithi067.stderr: 12500K .......... .......... .......... .......... .......... 17% 246M 0s 2019-12-09T20:48:00.828 INFO:teuthology.orchestra.run.smithi067.stderr: 12550K .......... .......... .......... .......... .......... 17% 224M 0s 2019-12-09T20:48:00.828 INFO:teuthology.orchestra.run.smithi067.stderr: 12600K .......... .......... .......... .......... .......... 17% 274M 0s 2019-12-09T20:48:00.828 INFO:teuthology.orchestra.run.smithi067.stderr: 12650K .......... .......... .......... .......... .......... 17% 271M 0s 2019-12-09T20:48:00.828 INFO:teuthology.orchestra.run.smithi067.stderr: 12700K .......... .......... .......... .......... .......... 17% 262M 0s 2019-12-09T20:48:00.829 INFO:teuthology.orchestra.run.smithi067.stderr: 12750K .......... .......... .......... .......... .......... 17% 236M 0s 2019-12-09T20:48:00.829 INFO:teuthology.orchestra.run.smithi067.stderr: 12800K .......... .......... .......... .......... .......... 17% 252M 0s 2019-12-09T20:48:00.829 INFO:teuthology.orchestra.run.smithi067.stderr: 12850K .......... .......... .......... .......... .......... 17% 256M 0s 2019-12-09T20:48:00.829 INFO:teuthology.orchestra.run.smithi067.stderr: 12900K .......... .......... .......... .......... .......... 17% 264M 0s 2019-12-09T20:48:00.829 INFO:teuthology.orchestra.run.smithi067.stderr: 12950K .......... .......... .......... .......... .......... 17% 214M 0s 2019-12-09T20:48:00.829 INFO:teuthology.orchestra.run.smithi067.stderr: 13000K .......... .......... .......... .......... .......... 17% 257M 0s 2019-12-09T20:48:00.829 INFO:teuthology.orchestra.run.smithi067.stderr: 13050K .......... .......... .......... .......... .......... 17% 265M 0s 2019-12-09T20:48:00.830 INFO:teuthology.orchestra.run.smithi067.stderr: 13100K .......... .......... .......... .......... .......... 17% 268M 0s 2019-12-09T20:48:00.830 INFO:teuthology.orchestra.run.smithi067.stderr: 13150K .......... .......... .......... .......... .......... 17% 225M 0s 2019-12-09T20:48:00.830 INFO:teuthology.orchestra.run.smithi067.stderr: 13200K .......... .......... .......... .......... .......... 18% 272M 0s 2019-12-09T20:48:00.830 INFO:teuthology.orchestra.run.smithi067.stderr: 13250K .......... .......... .......... .......... .......... 18% 284M 0s 2019-12-09T20:48:00.830 INFO:teuthology.orchestra.run.smithi067.stderr: 13300K .......... .......... .......... .......... .......... 18% 255M 0s 2019-12-09T20:48:00.830 INFO:teuthology.orchestra.run.smithi067.stderr: 13350K .......... .......... .......... .......... .......... 18% 214M 0s 2019-12-09T20:48:00.830 INFO:teuthology.orchestra.run.smithi067.stderr: 13400K .......... .......... .......... .......... .......... 18% 263M 0s 2019-12-09T20:48:00.831 INFO:teuthology.orchestra.run.smithi067.stderr: 13450K .......... .......... .......... .......... .......... 18% 247M 0s 2019-12-09T20:48:00.831 INFO:teuthology.orchestra.run.smithi067.stderr: 13500K .......... .......... .......... .......... .......... 18% 269M 0s 2019-12-09T20:48:00.831 INFO:teuthology.orchestra.run.smithi067.stderr: 13550K .......... .......... .......... .......... .......... 18% 236M 0s 2019-12-09T20:48:00.831 INFO:teuthology.orchestra.run.smithi067.stderr: 13600K .......... .......... .......... .......... .......... 18% 255M 0s 2019-12-09T20:48:00.831 INFO:teuthology.orchestra.run.smithi067.stderr: 13650K .......... .......... .......... .......... .......... 18% 264M 0s 2019-12-09T20:48:00.831 INFO:teuthology.orchestra.run.smithi067.stderr: 13700K .......... .......... .......... .......... .......... 18% 274M 0s 2019-12-09T20:48:00.831 INFO:teuthology.orchestra.run.smithi067.stderr: 13750K .......... .......... .......... .......... .......... 18% 223M 0s 2019-12-09T20:48:00.831 INFO:teuthology.orchestra.run.smithi067.stderr: 13800K .......... .......... .......... .......... .......... 18% 264M 0s 2019-12-09T20:48:00.832 INFO:teuthology.orchestra.run.smithi067.stderr: 13850K .......... .......... .......... .......... .......... 18% 264M 0s 2019-12-09T20:48:00.832 INFO:teuthology.orchestra.run.smithi067.stderr: 13900K .......... .......... .......... .......... .......... 18% 262M 0s 2019-12-09T20:48:00.832 INFO:teuthology.orchestra.run.smithi067.stderr: 13950K .......... .......... .......... .......... .......... 19% 187M 0s 2019-12-09T20:48:00.832 INFO:teuthology.orchestra.run.smithi067.stderr: 14000K .......... .......... .......... .......... .......... 19% 248M 0s 2019-12-09T20:48:00.832 INFO:teuthology.orchestra.run.smithi067.stderr: 14050K .......... .......... .......... .......... .......... 19% 245M 0s 2019-12-09T20:48:00.832 INFO:teuthology.orchestra.run.smithi067.stderr: 14100K .......... .......... .......... .......... .......... 19% 274M 0s 2019-12-09T20:48:00.832 INFO:teuthology.orchestra.run.smithi067.stderr: 14150K .......... .......... .......... .......... .......... 19% 249M 0s 2019-12-09T20:48:00.833 INFO:teuthology.orchestra.run.smithi067.stderr: 14200K .......... .......... .......... .......... .......... 19% 274M 0s 2019-12-09T20:48:00.833 INFO:teuthology.orchestra.run.smithi067.stderr: 14250K .......... .......... .......... .......... .......... 19% 277M 0s 2019-12-09T20:48:00.833 INFO:teuthology.orchestra.run.smithi067.stderr: 14300K .......... .......... .......... .......... .......... 19% 262M 0s 2019-12-09T20:48:00.833 INFO:teuthology.orchestra.run.smithi067.stderr: 14350K .......... .......... .......... .......... .......... 19% 236M 0s 2019-12-09T20:48:00.835 INFO:teuthology.orchestra.run.smithi067.stderr: 14400K .......... .......... .......... .......... .......... 19% 289M 0s 2019-12-09T20:48:00.835 INFO:teuthology.orchestra.run.smithi067.stderr: 14450K .......... .......... .......... .......... .......... 19% 296M 0s 2019-12-09T20:48:00.836 INFO:teuthology.orchestra.run.smithi067.stderr: 14500K .......... .......... .......... .......... .......... 19% 294M 0s 2019-12-09T20:48:00.836 INFO:teuthology.orchestra.run.smithi067.stderr: 14550K .......... .......... .......... .......... .......... 19% 246M 0s 2019-12-09T20:48:00.836 INFO:teuthology.orchestra.run.smithi067.stderr: 14600K .......... .......... .......... .......... .......... 19% 292M 0s 2019-12-09T20:48:00.836 INFO:teuthology.orchestra.run.smithi067.stderr: 14650K .......... .......... .......... .......... .......... 19% 287M 0s 2019-12-09T20:48:00.836 INFO:teuthology.orchestra.run.smithi067.stderr: 14700K .......... .......... .......... .......... .......... 20% 294M 0s 2019-12-09T20:48:00.836 INFO:teuthology.orchestra.run.smithi067.stderr: 14750K .......... .......... .......... .......... .......... 20% 244M 0s 2019-12-09T20:48:00.836 INFO:teuthology.orchestra.run.smithi067.stderr: 14800K .......... .......... .......... .......... .......... 20% 261M 0s 2019-12-09T20:48:00.837 INFO:teuthology.orchestra.run.smithi067.stderr: 14850K .......... .......... .......... .......... .......... 20% 276M 0s 2019-12-09T20:48:00.837 INFO:teuthology.orchestra.run.smithi067.stderr: 14900K .......... .......... .......... .......... .......... 20% 282M 0s 2019-12-09T20:48:00.837 INFO:teuthology.orchestra.run.smithi067.stderr: 14950K .......... .......... .......... .......... .......... 20% 228M 0s 2019-12-09T20:48:00.837 INFO:teuthology.orchestra.run.smithi067.stderr: 15000K .......... .......... .......... .......... .......... 20% 270M 0s 2019-12-09T20:48:00.837 INFO:teuthology.orchestra.run.smithi067.stderr: 15050K .......... .......... .......... .......... .......... 20% 294M 0s 2019-12-09T20:48:00.837 INFO:teuthology.orchestra.run.smithi067.stderr: 15100K .......... .......... .......... .......... .......... 20% 298M 0s 2019-12-09T20:48:00.837 INFO:teuthology.orchestra.run.smithi067.stderr: 15150K .......... .......... .......... .......... .......... 20% 263M 0s 2019-12-09T20:48:00.838 INFO:teuthology.orchestra.run.smithi067.stderr: 15200K .......... .......... .......... .......... .......... 20% 275M 0s 2019-12-09T20:48:00.838 INFO:teuthology.orchestra.run.smithi067.stderr: 15250K .......... .......... .......... .......... .......... 20% 287M 0s 2019-12-09T20:48:00.838 INFO:teuthology.orchestra.run.smithi067.stderr: 15300K .......... .......... .......... .......... .......... 20% 283M 0s 2019-12-09T20:48:00.838 INFO:teuthology.orchestra.run.smithi067.stderr: 15350K .......... .......... .......... .......... .......... 20% 236M 0s 2019-12-09T20:48:00.838 INFO:teuthology.orchestra.run.smithi067.stderr: 15400K .......... .......... .......... .......... .......... 21% 292M 0s 2019-12-09T20:48:00.838 INFO:teuthology.orchestra.run.smithi067.stderr: 15450K .......... .......... .......... .......... .......... 21% 284M 0s 2019-12-09T20:48:00.838 INFO:teuthology.orchestra.run.smithi067.stderr: 15500K .......... .......... .......... .......... .......... 21% 271M 0s 2019-12-09T20:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr: 15550K .......... .......... .......... .......... .......... 21% 266M 0s 2019-12-09T20:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr: 15600K .......... .......... .......... .......... .......... 21% 68.8M 0s 2019-12-09T20:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr: 15650K .......... .......... .......... .......... .......... 21% 254M 0s 2019-12-09T20:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr: 15700K .......... .......... .......... .......... .......... 21% 266M 0s 2019-12-09T20:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr: 15750K .......... .......... .......... .......... .......... 21% 239M 0s 2019-12-09T20:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr: 15800K .......... .......... .......... .......... .......... 21% 292M 0s 2019-12-09T20:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr: 15850K .......... .......... .......... .......... .......... 21% 8.34M 0s 2019-12-09T20:48:00.839 INFO:teuthology.orchestra.run.smithi067.stderr: 15900K .......... .......... .......... .......... .......... 21% 253M 0s 2019-12-09T20:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr: 15950K .......... .......... .......... .......... .......... 21% 224M 0s 2019-12-09T20:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr: 16000K .......... .......... .......... .......... .......... 21% 292M 0s 2019-12-09T20:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr: 16050K .......... .......... .......... .......... .......... 21% 292M 0s 2019-12-09T20:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr: 16100K .......... .......... .......... .......... .......... 21% 283M 0s 2019-12-09T20:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr: 16150K .......... .......... .......... .......... .......... 22% 167M 0s 2019-12-09T20:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr: 16200K .......... .......... .......... .......... .......... 22% 190M 0s 2019-12-09T20:48:00.840 INFO:teuthology.orchestra.run.smithi067.stderr: 16250K .......... .......... .......... .......... .......... 22% 200M 0s 2019-12-09T20:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr: 16300K .......... .......... .......... .......... .......... 22% 196M 0s 2019-12-09T20:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr: 16350K .......... .......... .......... .......... .......... 22% 164M 0s 2019-12-09T20:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr: 16400K .......... .......... .......... .......... .......... 22% 200M 0s 2019-12-09T20:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr: 16450K .......... .......... .......... .......... .......... 22% 201M 0s 2019-12-09T20:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr: 16500K .......... .......... .......... .......... .......... 22% 203M 0s 2019-12-09T20:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr: 16550K .......... .......... .......... .......... .......... 22% 170M 0s 2019-12-09T20:48:00.841 INFO:teuthology.orchestra.run.smithi067.stderr: 16600K .......... .......... .......... .......... .......... 22% 200M 0s 2019-12-09T20:48:00.842 INFO:teuthology.orchestra.run.smithi067.stderr: 16650K .......... .......... .......... .......... .......... 22% 212M 0s 2019-12-09T20:48:00.842 INFO:teuthology.orchestra.run.smithi067.stderr: 16700K .......... .......... .......... .......... .......... 22% 195M 0s 2019-12-09T20:48:00.842 INFO:teuthology.orchestra.run.smithi067.stderr: 16750K .......... .......... .......... .......... .......... 22% 188M 0s 2019-12-09T20:48:00.842 INFO:teuthology.orchestra.run.smithi067.stderr: 16800K .......... .......... .......... .......... .......... 22% 194M 0s 2019-12-09T20:48:00.842 INFO:teuthology.orchestra.run.smithi067.stderr: 16850K .......... .......... .......... .......... .......... 22% 188M 0s 2019-12-09T20:48:00.842 INFO:teuthology.orchestra.run.smithi067.stderr: 16900K .......... .......... .......... .......... .......... 23% 206M 0s 2019-12-09T20:48:00.842 INFO:teuthology.orchestra.run.smithi067.stderr: 16950K .......... .......... .......... .......... .......... 23% 113M 0s 2019-12-09T20:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr: 17000K .......... .......... .......... .......... .......... 23% 51.4M 0s 2019-12-09T20:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr: 17050K .......... .......... .......... .......... .......... 23% 195M 0s 2019-12-09T20:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr: 17100K .......... .......... .......... .......... .......... 23% 209M 0s 2019-12-09T20:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr: 17150K .......... .......... .......... .......... .......... 23% 105M 0s 2019-12-09T20:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr: 17200K .......... .......... .......... .......... .......... 23% 197M 0s 2019-12-09T20:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr: 17250K .......... .......... .......... .......... .......... 23% 209M 0s 2019-12-09T20:48:00.843 INFO:teuthology.orchestra.run.smithi067.stderr: 17300K .......... .......... .......... .......... .......... 23% 24.6M 0s 2019-12-09T20:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr: 17350K .......... .......... .......... .......... .......... 23% 157M 0s 2019-12-09T20:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr: 17400K .......... .......... .......... .......... .......... 23% 192M 0s 2019-12-09T20:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr: 17450K .......... .......... .......... .......... .......... 23% 188M 0s 2019-12-09T20:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr: 17500K .......... .......... .......... .......... .......... 23% 210M 0s 2019-12-09T20:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr: 17550K .......... .......... .......... .......... .......... 23% 179M 0s 2019-12-09T20:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr: 17600K .......... .......... .......... .......... .......... 23% 188M 0s 2019-12-09T20:48:00.844 INFO:teuthology.orchestra.run.smithi067.stderr: 17650K .......... .......... .......... .......... .......... 24% 191M 0s 2019-12-09T20:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr: 17700K .......... .......... .......... .......... .......... 24% 213M 0s 2019-12-09T20:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr: 17750K .......... .......... .......... .......... .......... 24% 177M 0s 2019-12-09T20:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr: 17800K .......... .......... .......... .......... .......... 24% 204M 0s 2019-12-09T20:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr: 17850K .......... .......... .......... .......... .......... 24% 34.3M 0s 2019-12-09T20:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr: 17900K .......... .......... .......... .......... .......... 24% 199M 0s 2019-12-09T20:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr: 17950K .......... .......... .......... .......... .......... 24% 164M 0s 2019-12-09T20:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr: 18000K .......... .......... .......... .......... .......... 24% 204M 0s 2019-12-09T20:48:00.845 INFO:teuthology.orchestra.run.smithi067.stderr: 18050K .......... .......... .......... .......... .......... 24% 201M 0s 2019-12-09T20:48:00.846 INFO:teuthology.orchestra.run.smithi067.stderr: 18100K .......... .......... .......... .......... .......... 24% 223M 0s 2019-12-09T20:48:00.846 INFO:teuthology.orchestra.run.smithi067.stderr: 18150K .......... .......... .......... .......... .......... 24% 193M 0s 2019-12-09T20:48:00.846 INFO:teuthology.orchestra.run.smithi067.stderr: 18200K .......... .......... .......... .......... .......... 24% 213M 0s 2019-12-09T20:48:00.846 INFO:teuthology.orchestra.run.smithi067.stderr: 18250K .......... .......... .......... .......... .......... 24% 210M 0s 2019-12-09T20:48:00.846 INFO:teuthology.orchestra.run.smithi067.stderr: 18300K .......... .......... .......... .......... .......... 24% 207M 0s 2019-12-09T20:48:00.846 INFO:teuthology.orchestra.run.smithi067.stderr: 18350K .......... .......... .......... .......... .......... 25% 187M 0s 2019-12-09T20:48:00.846 INFO:teuthology.orchestra.run.smithi067.stderr: 18400K .......... .......... .......... .......... .......... 25% 208M 0s 2019-12-09T20:48:00.847 INFO:teuthology.orchestra.run.smithi067.stderr: 18450K .......... .......... .......... .......... .......... 25% 204M 0s 2019-12-09T20:48:00.847 INFO:teuthology.orchestra.run.smithi067.stderr: 18500K .......... .......... .......... .......... .......... 25% 212M 0s 2019-12-09T20:48:00.847 INFO:teuthology.orchestra.run.smithi067.stderr: 18550K .......... .......... .......... .......... .......... 25% 173M 0s 2019-12-09T20:48:00.847 INFO:teuthology.orchestra.run.smithi067.stderr: 18600K .......... .......... .......... .......... .......... 25% 197M 0s 2019-12-09T20:48:00.847 INFO:teuthology.orchestra.run.smithi067.stderr: 18650K .......... .......... .......... .......... .......... 25% 212M 0s 2019-12-09T20:48:00.847 INFO:teuthology.orchestra.run.smithi067.stderr: 18700K .......... .......... .......... .......... .......... 25% 218M 0s 2019-12-09T20:48:00.847 INFO:teuthology.orchestra.run.smithi067.stderr: 18750K .......... .......... .......... .......... .......... 25% 179M 0s 2019-12-09T20:48:00.848 INFO:teuthology.orchestra.run.smithi067.stderr: 18800K .......... .......... .......... .......... .......... 25% 193M 0s 2019-12-09T20:48:00.848 INFO:teuthology.orchestra.run.smithi067.stderr: 18850K .......... .......... .......... .......... .......... 25% 201M 0s 2019-12-09T20:48:00.848 INFO:teuthology.orchestra.run.smithi067.stderr: 18900K .......... .......... .......... .......... .......... 25% 211M 0s 2019-12-09T20:48:00.848 INFO:teuthology.orchestra.run.smithi067.stderr: 18950K .......... .......... .......... .......... .......... 25% 200M 0s 2019-12-09T20:48:00.848 INFO:teuthology.orchestra.run.smithi067.stderr: 19000K .......... .......... .......... .......... .......... 25% 210M 0s 2019-12-09T20:48:00.848 INFO:teuthology.orchestra.run.smithi067.stderr: 19050K .......... .......... .......... .......... .......... 25% 206M 0s 2019-12-09T20:48:00.848 INFO:teuthology.orchestra.run.smithi067.stderr: 19100K .......... .......... .......... .......... .......... 26% 221M 0s 2019-12-09T20:48:00.849 INFO:teuthology.orchestra.run.smithi067.stderr: 19150K .......... .......... .......... .......... .......... 26% 184M 0s 2019-12-09T20:48:00.849 INFO:teuthology.orchestra.run.smithi067.stderr: 19200K .......... .......... .......... .......... .......... 26% 144M 0s 2019-12-09T20:48:00.849 INFO:teuthology.orchestra.run.smithi067.stderr: 19250K .......... .......... .......... .......... .......... 26% 206M 0s 2019-12-09T20:48:00.849 INFO:teuthology.orchestra.run.smithi067.stderr: 19300K .......... .......... .......... .......... .......... 26% 204M 0s 2019-12-09T20:48:00.849 INFO:teuthology.orchestra.run.smithi067.stderr: 19350K .......... .......... .......... .......... .......... 26% 180M 0s 2019-12-09T20:48:00.849 INFO:teuthology.orchestra.run.smithi067.stderr: 19400K .......... .......... .......... .......... .......... 26% 211M 0s 2019-12-09T20:48:00.849 INFO:teuthology.orchestra.run.smithi067.stderr: 19450K .......... .......... .......... .......... .......... 26% 198M 0s 2019-12-09T20:48:00.849 INFO:teuthology.orchestra.run.smithi067.stderr: 19500K .......... .......... .......... .......... .......... 26% 210M 0s 2019-12-09T20:48:00.850 INFO:teuthology.orchestra.run.smithi067.stderr: 19550K .......... .......... .......... .......... .......... 26% 189M 0s 2019-12-09T20:48:00.850 INFO:teuthology.orchestra.run.smithi067.stderr: 19600K .......... .......... .......... .......... .......... 26% 209M 0s 2019-12-09T20:48:00.850 INFO:teuthology.orchestra.run.smithi067.stderr: 19650K .......... .......... .......... .......... .......... 26% 209M 0s 2019-12-09T20:48:00.850 INFO:teuthology.orchestra.run.smithi067.stderr: 19700K .......... .......... .......... .......... .......... 26% 215M 0s 2019-12-09T20:48:00.850 INFO:teuthology.orchestra.run.smithi067.stderr: 19750K .......... .......... .......... .......... .......... 26% 191M 0s 2019-12-09T20:48:00.850 INFO:teuthology.orchestra.run.smithi067.stderr: 19800K .......... .......... .......... .......... .......... 26% 217M 0s 2019-12-09T20:48:00.850 INFO:teuthology.orchestra.run.smithi067.stderr: 19850K .......... .......... .......... .......... .......... 27% 223M 0s 2019-12-09T20:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr: 19900K .......... .......... .......... .......... .......... 27% 226M 0s 2019-12-09T20:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr: 19950K .......... .......... .......... .......... .......... 27% 25.9M 0s 2019-12-09T20:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr: 20000K .......... .......... .......... .......... .......... 27% 212M 0s 2019-12-09T20:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr: 20050K .......... .......... .......... .......... .......... 27% 210M 0s 2019-12-09T20:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr: 20100K .......... .......... .......... .......... .......... 27% 215M 0s 2019-12-09T20:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr: 20150K .......... .......... .......... .......... .......... 27% 194M 0s 2019-12-09T20:48:00.851 INFO:teuthology.orchestra.run.smithi067.stderr: 20200K .......... .......... .......... .......... .......... 27% 208M 0s 2019-12-09T20:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr: 20250K .......... .......... .......... .......... .......... 27% 217M 0s 2019-12-09T20:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr: 20300K .......... .......... .......... .......... .......... 27% 210M 0s 2019-12-09T20:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr: 20350K .......... .......... .......... .......... .......... 27% 171M 0s 2019-12-09T20:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr: 20400K .......... .......... .......... .......... .......... 27% 224M 0s 2019-12-09T20:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr: 20450K .......... .......... .......... .......... .......... 27% 221M 0s 2019-12-09T20:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr: 20500K .......... .......... .......... .......... .......... 27% 225M 0s 2019-12-09T20:48:00.852 INFO:teuthology.orchestra.run.smithi067.stderr: 20550K .......... .......... .......... .......... .......... 28% 194M 0s 2019-12-09T20:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr: 20600K .......... .......... .......... .......... .......... 28% 215M 0s 2019-12-09T20:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr: 20650K .......... .......... .......... .......... .......... 28% 196M 0s 2019-12-09T20:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr: 20700K .......... .......... .......... .......... .......... 28% 222M 0s 2019-12-09T20:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr: 20750K .......... .......... .......... .......... .......... 28% 183M 0s 2019-12-09T20:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr: 20800K .......... .......... .......... .......... .......... 28% 223M 0s 2019-12-09T20:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr: 20850K .......... .......... .......... .......... .......... 28% 227M 0s 2019-12-09T20:48:00.853 INFO:teuthology.orchestra.run.smithi067.stderr: 20900K .......... .......... .......... .......... .......... 28% 223M 0s 2019-12-09T20:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr: 20950K .......... .......... .......... .......... .......... 28% 206M 0s 2019-12-09T20:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr: 21000K .......... .......... .......... .......... .......... 28% 303M 0s 2019-12-09T20:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr: 21050K .......... .......... .......... .......... .......... 28% 295M 0s 2019-12-09T20:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr: 21100K .......... .......... .......... .......... .......... 28% 316M 0s 2019-12-09T20:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr: 21150K .......... .......... .......... .......... .......... 28% 275M 0s 2019-12-09T20:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr: 21200K .......... .......... .......... .......... .......... 28% 326M 0s 2019-12-09T20:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr: 21250K .......... .......... .......... .......... .......... 28% 336M 0s 2019-12-09T20:48:00.854 INFO:teuthology.orchestra.run.smithi067.stderr: 21300K .......... .......... .......... .......... .......... 29% 324M 0s 2019-12-09T20:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr: 21350K .......... .......... .......... .......... .......... 29% 286M 0s 2019-12-09T20:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr: 21400K .......... .......... .......... .......... .......... 29% 313M 0s 2019-12-09T20:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr: 21450K .......... .......... .......... .......... .......... 29% 330M 0s 2019-12-09T20:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr: 21500K .......... .......... .......... .......... .......... 29% 345M 0s 2019-12-09T20:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr: 21550K .......... .......... .......... .......... .......... 29% 249M 0s 2019-12-09T20:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr: 21600K .......... .......... .......... .......... .......... 29% 342M 0s 2019-12-09T20:48:00.855 INFO:teuthology.orchestra.run.smithi067.stderr: 21650K .......... .......... .......... .......... .......... 29% 355M 0s 2019-12-09T20:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr: 21700K .......... .......... .......... .......... .......... 29% 361M 0s 2019-12-09T20:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr: 21750K .......... .......... .......... .......... .......... 29% 220M 0s 2019-12-09T20:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr: 21800K .......... .......... .......... .......... .......... 29% 320M 0s 2019-12-09T20:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr: 21850K .......... .......... .......... .......... .......... 29% 345M 0s 2019-12-09T20:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr: 21900K .......... .......... .......... .......... .......... 29% 30.8M 0s 2019-12-09T20:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr: 21950K .......... .......... .......... .......... .......... 29% 229M 0s 2019-12-09T20:48:00.856 INFO:teuthology.orchestra.run.smithi067.stderr: 22000K .......... .......... .......... .......... .......... 29% 290M 0s 2019-12-09T20:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr: 22050K .......... .......... .......... .......... .......... 30% 317M 0s 2019-12-09T20:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr: 22100K .......... .......... .......... .......... .......... 30% 325M 0s 2019-12-09T20:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr: 22150K .......... .......... .......... .......... .......... 30% 267M 0s 2019-12-09T20:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr: 22200K .......... .......... .......... .......... .......... 30% 323M 0s 2019-12-09T20:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr: 22250K .......... .......... .......... .......... .......... 30% 327M 0s 2019-12-09T20:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr: 22300K .......... .......... .......... .......... .......... 30% 317M 0s 2019-12-09T20:48:00.857 INFO:teuthology.orchestra.run.smithi067.stderr: 22350K .......... .......... .......... .......... .......... 30% 313M 0s 2019-12-09T20:48:00.858 INFO:teuthology.orchestra.run.smithi067.stderr: 22400K .......... .......... .......... .......... .......... 30% 339M 0s 2019-12-09T20:48:00.858 INFO:teuthology.orchestra.run.smithi067.stderr: 22450K .......... .......... .......... .......... .......... 30% 351M 0s 2019-12-09T20:48:00.858 INFO:teuthology.orchestra.run.smithi067.stderr: 22500K .......... .......... .......... .......... .......... 30% 87.6M 0s 2019-12-09T20:48:00.858 INFO:teuthology.orchestra.run.smithi067.stderr: 22550K .......... .......... .......... .......... .......... 30% 290M 0s 2019-12-09T20:48:00.858 INFO:teuthology.orchestra.run.smithi067.stderr: 22600K .......... .......... .......... .......... .......... 30% 331M 0s 2019-12-09T20:48:00.858 INFO:teuthology.orchestra.run.smithi067.stderr: 22650K .......... .......... .......... .......... .......... 30% 320M 0s 2019-12-09T20:48:00.858 INFO:teuthology.orchestra.run.smithi067.stderr: 22700K .......... .......... .......... .......... .......... 30% 353M 0s 2019-12-09T20:48:00.859 INFO:teuthology.orchestra.run.smithi067.stderr: 22750K .......... .......... .......... .......... .......... 30% 48.2M 0s 2019-12-09T20:48:00.859 INFO:teuthology.orchestra.run.smithi067.stderr: 22800K .......... .......... .......... .......... .......... 31% 288M 0s 2019-12-09T20:48:00.859 INFO:teuthology.orchestra.run.smithi067.stderr: 22850K .......... .......... .......... .......... .......... 31% 332M 0s 2019-12-09T20:48:00.859 INFO:teuthology.orchestra.run.smithi067.stderr: 22900K .......... .......... .......... .......... .......... 31% 315M 0s 2019-12-09T20:48:00.859 INFO:teuthology.orchestra.run.smithi067.stderr: 22950K .......... .......... .......... .......... .......... 31% 295M 0s 2019-12-09T20:48:00.859 INFO:teuthology.orchestra.run.smithi067.stderr: 23000K .......... .......... .......... .......... .......... 31% 82.5M 0s 2019-12-09T20:48:00.859 INFO:teuthology.orchestra.run.smithi067.stderr: 23050K .......... .......... .......... .......... .......... 31% 226M 0s 2019-12-09T20:48:00.860 INFO:teuthology.orchestra.run.smithi067.stderr: 23100K .......... .......... .......... .......... .......... 31% 336M 0s 2019-12-09T20:48:00.860 INFO:teuthology.orchestra.run.smithi067.stderr: 23150K .......... .......... .......... .......... .......... 31% 18.2M 0s 2019-12-09T20:48:00.860 INFO:teuthology.orchestra.run.smithi067.stderr: 23200K .......... .......... .......... .......... .......... 31% 268M 0s 2019-12-09T20:48:00.860 INFO:teuthology.orchestra.run.smithi067.stderr: 23250K .......... .......... .......... .......... .......... 31% 271M 0s 2019-12-09T20:48:00.860 INFO:teuthology.orchestra.run.smithi067.stderr: 23300K .......... .......... .......... .......... .......... 31% 283M 0s 2019-12-09T20:48:00.860 INFO:teuthology.orchestra.run.smithi067.stderr: 23350K .......... .......... .......... .......... .......... 31% 267M 0s 2019-12-09T20:48:00.860 INFO:teuthology.orchestra.run.smithi067.stderr: 23400K .......... .......... .......... .......... .......... 31% 342M 0s 2019-12-09T20:48:00.861 INFO:teuthology.orchestra.run.smithi067.stderr: 23450K .......... .......... .......... .......... .......... 31% 329M 0s 2019-12-09T20:48:00.861 INFO:teuthology.orchestra.run.smithi067.stderr: 23500K .......... .......... .......... .......... .......... 32% 323M 0s 2019-12-09T20:48:00.861 INFO:teuthology.orchestra.run.smithi067.stderr: 23550K .......... .......... .......... .......... .......... 32% 277M 0s 2019-12-09T20:48:00.861 INFO:teuthology.orchestra.run.smithi067.stderr: 23600K .......... .......... .......... .......... .......... 32% 308M 0s 2019-12-09T20:48:00.861 INFO:teuthology.orchestra.run.smithi067.stderr: 23650K .......... .......... .......... .......... .......... 32% 294M 0s 2019-12-09T20:48:00.861 INFO:teuthology.orchestra.run.smithi067.stderr: 23700K .......... .......... .......... .......... .......... 32% 368M 0s 2019-12-09T20:48:00.861 INFO:teuthology.orchestra.run.smithi067.stderr: 23750K .......... .......... .......... .......... .......... 32% 303M 0s 2019-12-09T20:48:00.861 INFO:teuthology.orchestra.run.smithi067.stderr: 23800K .......... .......... .......... .......... .......... 32% 373M 0s 2019-12-09T20:48:00.862 INFO:teuthology.orchestra.run.smithi067.stderr: 23850K .......... .......... .......... .......... .......... 32% 371M 0s 2019-12-09T20:48:00.862 INFO:teuthology.orchestra.run.smithi067.stderr: 23900K .......... .......... .......... .......... .......... 32% 57.8M 0s 2019-12-09T20:48:00.862 INFO:teuthology.orchestra.run.smithi067.stderr: 23950K .......... .......... .......... .......... .......... 32% 241M 0s 2019-12-09T20:48:00.862 INFO:teuthology.orchestra.run.smithi067.stderr: 24000K .......... .......... .......... .......... .......... 32% 296M 0s 2019-12-09T20:48:00.862 INFO:teuthology.orchestra.run.smithi067.stderr: 24050K .......... .......... .......... .......... .......... 32% 290M 0s 2019-12-09T20:48:00.862 INFO:teuthology.orchestra.run.smithi067.stderr: 24100K .......... .......... .......... .......... .......... 32% 316M 0s 2019-12-09T20:48:00.862 INFO:teuthology.orchestra.run.smithi067.stderr: 24150K .......... .......... .......... .......... .......... 32% 288M 0s 2019-12-09T20:48:00.863 INFO:teuthology.orchestra.run.smithi067.stderr: 24200K .......... .......... .......... .......... .......... 32% 359M 0s 2019-12-09T20:48:00.863 INFO:teuthology.orchestra.run.smithi067.stderr: 24250K .......... .......... .......... .......... .......... 33% 365M 0s 2019-12-09T20:48:00.863 INFO:teuthology.orchestra.run.smithi067.stderr: 24300K .......... .......... .......... .......... .......... 33% 99.9M 0s 2019-12-09T20:48:00.863 INFO:teuthology.orchestra.run.smithi067.stderr: 24350K .......... .......... .......... .......... .......... 33% 282M 0s 2019-12-09T20:48:00.863 INFO:teuthology.orchestra.run.smithi067.stderr: 24400K .......... .......... .......... .......... .......... 33% 352M 0s 2019-12-09T20:48:00.863 INFO:teuthology.orchestra.run.smithi067.stderr: 24450K .......... .......... .......... .......... .......... 33% 345M 0s 2019-12-09T20:48:00.863 INFO:teuthology.orchestra.run.smithi067.stderr: 24500K .......... .......... .......... .......... .......... 33% 365M 0s 2019-12-09T20:48:00.864 INFO:teuthology.orchestra.run.smithi067.stderr: 24550K .......... .......... .......... .......... .......... 33% 50.1M 0s 2019-12-09T20:48:00.864 INFO:teuthology.orchestra.run.smithi067.stderr: 24600K .......... .......... .......... .......... .......... 33% 232M 0s 2019-12-09T20:48:00.864 INFO:teuthology.orchestra.run.smithi067.stderr: 24650K .......... .......... .......... .......... .......... 33% 316M 0s 2019-12-09T20:48:00.864 INFO:teuthology.orchestra.run.smithi067.stderr: 24700K .......... .......... .......... .......... .......... 33% 327M 0s 2019-12-09T20:48:00.864 INFO:teuthology.orchestra.run.smithi067.stderr: 24750K .......... .......... .......... .......... .......... 33% 309M 0s 2019-12-09T20:48:00.864 INFO:teuthology.orchestra.run.smithi067.stderr: 24800K .......... .......... .......... .......... .......... 33% 100M 0s 2019-12-09T20:48:00.864 INFO:teuthology.orchestra.run.smithi067.stderr: 24850K .......... .......... .......... .......... .......... 33% 219M 0s 2019-12-09T20:48:00.865 INFO:teuthology.orchestra.run.smithi067.stderr: 24900K .......... .......... .......... .......... .......... 33% 328M 0s 2019-12-09T20:48:00.865 INFO:teuthology.orchestra.run.smithi067.stderr: 24950K .......... .......... .......... .......... .......... 33% 266M 0s 2019-12-09T20:48:00.865 INFO:teuthology.orchestra.run.smithi067.stderr: 25000K .......... .......... .......... .......... .......... 34% 318M 0s 2019-12-09T20:48:00.865 INFO:teuthology.orchestra.run.smithi067.stderr: 25050K .......... .......... .......... .......... .......... 34% 358M 0s 2019-12-09T20:48:00.865 INFO:teuthology.orchestra.run.smithi067.stderr: 25100K .......... .......... .......... .......... .......... 34% 376M 0s 2019-12-09T20:48:00.865 INFO:teuthology.orchestra.run.smithi067.stderr: 25150K .......... .......... .......... .......... .......... 34% 14.8M 0s 2019-12-09T20:48:00.865 INFO:teuthology.orchestra.run.smithi067.stderr: 25200K .......... .......... .......... .......... .......... 34% 145M 0s 2019-12-09T20:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr: 25250K .......... .......... .......... .......... .......... 34% 138M 0s 2019-12-09T20:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr: 25300K .......... .......... .......... .......... .......... 34% 147M 0s 2019-12-09T20:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr: 25350K .......... .......... .......... .......... .......... 34% 125M 0s 2019-12-09T20:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr: 25400K .......... .......... .......... .......... .......... 34% 154M 0s 2019-12-09T20:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr: 25450K .......... .......... .......... .......... .......... 34% 160M 0s 2019-12-09T20:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr: 25500K .......... .......... .......... .......... .......... 34% 158M 0s 2019-12-09T20:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr: 25550K .......... .......... .......... .......... .......... 34% 124M 0s 2019-12-09T20:48:00.866 INFO:teuthology.orchestra.run.smithi067.stderr: 25600K .......... .......... .......... .......... .......... 34% 150M 0s 2019-12-09T20:48:00.867 INFO:teuthology.orchestra.run.smithi067.stderr: 25650K .......... .......... .......... .......... .......... 34% 155M 0s 2019-12-09T20:48:00.867 INFO:teuthology.orchestra.run.smithi067.stderr: 25700K .......... .......... .......... .......... .......... 35% 157M 0s 2019-12-09T20:48:00.867 INFO:teuthology.orchestra.run.smithi067.stderr: 25750K .......... .......... .......... .......... .......... 35% 134M 0s 2019-12-09T20:48:00.867 INFO:teuthology.orchestra.run.smithi067.stderr: 25800K .......... .......... .......... .......... .......... 35% 144M 0s 2019-12-09T20:48:00.867 INFO:teuthology.orchestra.run.smithi067.stderr: 25850K .......... .......... .......... .......... .......... 35% 153M 0s 2019-12-09T20:48:00.867 INFO:teuthology.orchestra.run.smithi067.stderr: 25900K .......... .......... .......... .......... .......... 35% 163M 0s 2019-12-09T20:48:00.867 INFO:teuthology.orchestra.run.smithi067.stderr: 25950K .......... .......... .......... .......... .......... 35% 135M 0s 2019-12-09T20:48:00.868 INFO:teuthology.orchestra.run.smithi067.stderr: 26000K .......... .......... .......... .......... .......... 35% 156M 0s 2019-12-09T20:48:00.868 INFO:teuthology.orchestra.run.smithi067.stderr: 26050K .......... .......... .......... .......... .......... 35% 159M 0s 2019-12-09T20:48:00.868 INFO:teuthology.orchestra.run.smithi067.stderr: 26100K .......... .......... .......... .......... .......... 35% 110M 0s 2019-12-09T20:48:00.868 INFO:teuthology.orchestra.run.smithi067.stderr: 26150K .......... .......... .......... .......... .......... 35% 92.7M 0s 2019-12-09T20:48:00.868 INFO:teuthology.orchestra.run.smithi067.stderr: 26200K .......... .......... .......... .......... .......... 35% 111M 0s 2019-12-09T20:48:00.868 INFO:teuthology.orchestra.run.smithi067.stderr: 26250K .......... .......... .......... .......... .......... 35% 105M 0s 2019-12-09T20:48:00.868 INFO:teuthology.orchestra.run.smithi067.stderr: 26300K .......... .......... .......... .......... .......... 35% 112M 0s 2019-12-09T20:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr: 26350K .......... .......... .......... .......... .......... 35% 92.6M 0s 2019-12-09T20:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr: 26400K .......... .......... .......... .......... .......... 35% 113M 0s 2019-12-09T20:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr: 26450K .......... .......... .......... .......... .......... 36% 107M 0s 2019-12-09T20:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr: 26500K .......... .......... .......... .......... .......... 36% 120M 0s 2019-12-09T20:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr: 26550K .......... .......... .......... .......... .......... 36% 102M 0s 2019-12-09T20:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr: 26600K .......... .......... .......... .......... .......... 36% 117M 0s 2019-12-09T20:48:00.869 INFO:teuthology.orchestra.run.smithi067.stderr: 26650K .......... .......... .......... .......... .......... 36% 129M 0s 2019-12-09T20:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr: 26700K .......... .......... .......... .......... .......... 36% 130M 0s 2019-12-09T20:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr: 26750K .......... .......... .......... .......... .......... 36% 117M 0s 2019-12-09T20:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr: 26800K .......... .......... .......... .......... .......... 36% 135M 0s 2019-12-09T20:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr: 26850K .......... .......... .......... .......... .......... 36% 119M 0s 2019-12-09T20:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr: 26900K .......... .......... .......... .......... .......... 36% 128M 0s 2019-12-09T20:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr: 26950K .......... .......... .......... .......... .......... 36% 111M 0s 2019-12-09T20:48:00.870 INFO:teuthology.orchestra.run.smithi067.stderr: 27000K .......... .......... .......... .......... .......... 36% 120M 0s 2019-12-09T20:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr: 27050K .......... .......... .......... .......... .......... 36% 119M 0s 2019-12-09T20:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr: 27100K .......... .......... .......... .......... .......... 36% 120M 0s 2019-12-09T20:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr: 27150K .......... .......... .......... .......... .......... 36% 99.5M 0s 2019-12-09T20:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr: 27200K .......... .......... .......... .......... .......... 37% 115M 0s 2019-12-09T20:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr: 27250K .......... .......... .......... .......... .......... 37% 127M 0s 2019-12-09T20:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr: 27300K .......... .......... .......... .......... .......... 37% 126M 0s 2019-12-09T20:48:00.871 INFO:teuthology.orchestra.run.smithi067.stderr: 27350K .......... .......... .......... .......... .......... 37% 113M 0s 2019-12-09T20:48:00.872 INFO:teuthology.orchestra.run.smithi067.stderr: 27400K .......... .......... .......... .......... .......... 37% 129M 0s 2019-12-09T20:48:00.872 INFO:teuthology.orchestra.run.smithi067.stderr: 27450K .......... .......... .......... .......... .......... 37% 128M 0s 2019-12-09T20:48:00.872 INFO:teuthology.orchestra.run.smithi067.stderr: 27500K .......... .......... .......... .......... .......... 37% 134M 0s 2019-12-09T20:48:00.872 INFO:teuthology.orchestra.run.smithi067.stderr: 27550K .......... .......... .......... .......... .......... 37% 110M 0s 2019-12-09T20:48:00.872 INFO:teuthology.orchestra.run.smithi067.stderr: 27600K .......... .......... .......... .......... .......... 37% 135M 0s 2019-12-09T20:48:00.872 INFO:teuthology.orchestra.run.smithi067.stderr: 27650K .......... .......... .......... .......... .......... 37% 142M 0s 2019-12-09T20:48:00.872 INFO:teuthology.orchestra.run.smithi067.stderr: 27700K .......... .......... .......... .......... .......... 37% 128M 0s 2019-12-09T20:48:00.872 INFO:teuthology.orchestra.run.smithi067.stderr: 27750K .......... .......... .......... .......... .......... 37% 107M 0s 2019-12-09T20:48:00.873 INFO:teuthology.orchestra.run.smithi067.stderr: 27800K .......... .......... .......... .......... .......... 37% 135M 0s 2019-12-09T20:48:00.873 INFO:teuthology.orchestra.run.smithi067.stderr: 27850K .......... .......... .......... .......... .......... 37% 130M 0s 2019-12-09T20:48:00.873 INFO:teuthology.orchestra.run.smithi067.stderr: 27900K .......... .......... .......... .......... .......... 37% 138M 0s 2019-12-09T20:48:00.873 INFO:teuthology.orchestra.run.smithi067.stderr: 27950K .......... .......... .......... .......... .......... 38% 121M 0s 2019-12-09T20:48:00.873 INFO:teuthology.orchestra.run.smithi067.stderr: 28000K .......... .......... .......... .......... .......... 38% 132M 0s 2019-12-09T20:48:00.873 INFO:teuthology.orchestra.run.smithi067.stderr: 28050K .......... .......... .......... .......... .......... 38% 138M 0s 2019-12-09T20:48:00.873 INFO:teuthology.orchestra.run.smithi067.stderr: 28100K .......... .......... .......... .......... .......... 38% 129M 0s 2019-12-09T20:48:00.874 INFO:teuthology.orchestra.run.smithi067.stderr: 28150K .......... .......... .......... .......... .......... 38% 116M 0s 2019-12-09T20:48:00.874 INFO:teuthology.orchestra.run.smithi067.stderr: 28200K .......... .......... .......... .......... .......... 38% 139M 0s 2019-12-09T20:48:00.874 INFO:teuthology.orchestra.run.smithi067.stderr: 28250K .......... .......... .......... .......... .......... 38% 137M 0s 2019-12-09T20:48:00.874 INFO:teuthology.orchestra.run.smithi067.stderr: 28300K .......... .......... .......... .......... .......... 38% 132M 0s 2019-12-09T20:48:00.874 INFO:teuthology.orchestra.run.smithi067.stderr: 28350K .......... .......... .......... .......... .......... 38% 125M 0s 2019-12-09T20:48:00.874 INFO:teuthology.orchestra.run.smithi067.stderr: 28400K .......... .......... .......... .......... .......... 38% 134M 0s 2019-12-09T20:48:00.874 INFO:teuthology.orchestra.run.smithi067.stderr: 28450K .......... .......... .......... .......... .......... 38% 135M 0s 2019-12-09T20:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr: 28500K .......... .......... .......... .......... .......... 38% 134M 0s 2019-12-09T20:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr: 28550K .......... .......... .......... .......... .......... 38% 110M 0s 2019-12-09T20:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr: 28600K .......... .......... .......... .......... .......... 38% 133M 0s 2019-12-09T20:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr: 28650K .......... .......... .......... .......... .......... 39% 131M 0s 2019-12-09T20:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr: 28700K .......... .......... .......... .......... .......... 39% 144M 0s 2019-12-09T20:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr: 28750K .......... .......... .......... .......... .......... 39% 124M 0s 2019-12-09T20:48:00.875 INFO:teuthology.orchestra.run.smithi067.stderr: 28800K .......... .......... .......... .......... .......... 39% 134M 0s 2019-12-09T20:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr: 28850K .......... .......... .......... .......... .......... 39% 139M 0s 2019-12-09T20:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr: 28900K .......... .......... .......... .......... .......... 39% 133M 0s 2019-12-09T20:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr: 28950K .......... .......... .......... .......... .......... 39% 129M 0s 2019-12-09T20:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr: 29000K .......... .......... .......... .......... .......... 39% 134M 0s 2019-12-09T20:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr: 29050K .......... .......... .......... .......... .......... 39% 141M 0s 2019-12-09T20:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr: 29100K .......... .......... .......... .......... .......... 39% 146M 0s 2019-12-09T20:48:00.876 INFO:teuthology.orchestra.run.smithi067.stderr: 29150K .......... .......... .......... .......... .......... 39% 120M 0s 2019-12-09T20:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr: 29200K .......... .......... .......... .......... .......... 39% 147M 0s 2019-12-09T20:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr: 29250K .......... .......... .......... .......... .......... 39% 148M 0s 2019-12-09T20:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr: 29300K .......... .......... .......... .......... .......... 39% 144M 0s 2019-12-09T20:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr: 29350K .......... .......... .......... .......... .......... 39% 118M 0s 2019-12-09T20:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr: 29400K .......... .......... .......... .......... .......... 40% 147M 0s 2019-12-09T20:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr: 29450K .......... .......... .......... .......... .......... 40% 143M 0s 2019-12-09T20:48:00.877 INFO:teuthology.orchestra.run.smithi067.stderr: 29500K .......... .......... .......... .......... .......... 40% 143M 0s 2019-12-09T20:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr: 29550K .......... .......... .......... .......... .......... 40% 124M 0s 2019-12-09T20:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr: 29600K .......... .......... .......... .......... .......... 40% 146M 0s 2019-12-09T20:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr: 29650K .......... .......... .......... .......... .......... 40% 148M 0s 2019-12-09T20:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr: 29700K .......... .......... .......... .......... .......... 40% 146M 0s 2019-12-09T20:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr: 29750K .......... .......... .......... .......... .......... 40% 132M 0s 2019-12-09T20:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr: 29800K .......... .......... .......... .......... .......... 40% 142M 0s 2019-12-09T20:48:00.878 INFO:teuthology.orchestra.run.smithi067.stderr: 29850K .......... .......... .......... .......... .......... 40% 143M 0s 2019-12-09T20:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr: 29900K .......... .......... .......... .......... .......... 40% 140M 0s 2019-12-09T20:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr: 29950K .......... .......... .......... .......... .......... 40% 119M 0s 2019-12-09T20:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr: 30000K .......... .......... .......... .......... .......... 40% 140M 0s 2019-12-09T20:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr: 30050K .......... .......... .......... .......... .......... 40% 147M 0s 2019-12-09T20:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr: 30100K .......... .......... .......... .......... .......... 40% 149M 0s 2019-12-09T20:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr: 30150K .......... .......... .......... .......... .......... 41% 132M 0s 2019-12-09T20:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr: 30200K .......... .......... .......... .......... .......... 41% 149M 0s 2019-12-09T20:48:00.879 INFO:teuthology.orchestra.run.smithi067.stderr: 30250K .......... .......... .......... .......... .......... 41% 149M 0s 2019-12-09T20:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr: 30300K .......... .......... .......... .......... .......... 41% 149M 0s 2019-12-09T20:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr: 30350K .......... .......... .......... .......... .......... 41% 114M 0s 2019-12-09T20:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr: 30400K .......... .......... .......... .......... .......... 41% 152M 0s 2019-12-09T20:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr: 30450K .......... .......... .......... .......... .......... 41% 144M 0s 2019-12-09T20:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr: 30500K .......... .......... .......... .......... .......... 41% 148M 0s 2019-12-09T20:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr: 30550K .......... .......... .......... .......... .......... 41% 129M 0s 2019-12-09T20:48:00.880 INFO:teuthology.orchestra.run.smithi067.stderr: 30600K .......... .......... .......... .......... .......... 41% 137M 0s 2019-12-09T20:48:00.881 INFO:teuthology.orchestra.run.smithi067.stderr: 30650K .......... .......... .......... .......... .......... 41% 138M 0s 2019-12-09T20:48:00.881 INFO:teuthology.orchestra.run.smithi067.stderr: 30700K .......... .......... .......... .......... .......... 41% 151M 0s 2019-12-09T20:48:00.881 INFO:teuthology.orchestra.run.smithi067.stderr: 30750K .......... .......... .......... .......... .......... 41% 129M 0s 2019-12-09T20:48:00.881 INFO:teuthology.orchestra.run.smithi067.stderr: 30800K .......... .......... .......... .......... .......... 41% 152M 0s 2019-12-09T20:48:00.881 INFO:teuthology.orchestra.run.smithi067.stderr: 30850K .......... .......... .......... .......... .......... 42% 158M 0s 2019-12-09T20:48:00.881 INFO:teuthology.orchestra.run.smithi067.stderr: 30900K .......... .......... .......... .......... .......... 42% 143M 0s 2019-12-09T20:48:00.881 INFO:teuthology.orchestra.run.smithi067.stderr: 30950K .......... .......... .......... .......... .......... 42% 131M 0s 2019-12-09T20:48:00.882 INFO:teuthology.orchestra.run.smithi067.stderr: 31000K .......... .......... .......... .......... .......... 42% 148M 0s 2019-12-09T20:48:00.882 INFO:teuthology.orchestra.run.smithi067.stderr: 31050K .......... .......... .......... .......... .......... 42% 148M 0s 2019-12-09T20:48:00.882 INFO:teuthology.orchestra.run.smithi067.stderr: 31100K .......... .......... .......... .......... .......... 42% 156M 0s 2019-12-09T20:48:00.882 INFO:teuthology.orchestra.run.smithi067.stderr: 31150K .......... .......... .......... .......... .......... 42% 126M 0s 2019-12-09T20:48:00.882 INFO:teuthology.orchestra.run.smithi067.stderr: 31200K .......... .......... .......... .......... .......... 42% 154M 0s 2019-12-09T20:48:00.882 INFO:teuthology.orchestra.run.smithi067.stderr: 31250K .......... .......... .......... .......... .......... 42% 158M 0s 2019-12-09T20:48:00.882 INFO:teuthology.orchestra.run.smithi067.stderr: 31300K .......... .......... .......... .......... .......... 42% 156M 0s 2019-12-09T20:48:00.883 INFO:teuthology.orchestra.run.smithi067.stderr: 31350K .......... .......... .......... .......... .......... 42% 141M 0s 2019-12-09T20:48:00.883 INFO:teuthology.orchestra.run.smithi067.stderr: 31400K .......... .......... .......... .......... .......... 42% 155M 0s 2019-12-09T20:48:00.883 INFO:teuthology.orchestra.run.smithi067.stderr: 31450K .......... .......... .......... .......... .......... 42% 155M 0s 2019-12-09T20:48:00.883 INFO:teuthology.orchestra.run.smithi067.stderr: 31500K .......... .......... .......... .......... .......... 42% 145M 0s 2019-12-09T20:48:00.883 INFO:teuthology.orchestra.run.smithi067.stderr: 31550K .......... .......... .......... .......... .......... 42% 122M 0s 2019-12-09T20:48:00.883 INFO:teuthology.orchestra.run.smithi067.stderr: 31600K .......... .......... .......... .......... .......... 43% 152M 0s 2019-12-09T20:48:00.883 INFO:teuthology.orchestra.run.smithi067.stderr: 31650K .......... .......... .......... .......... .......... 43% 154M 0s 2019-12-09T20:48:00.883 INFO:teuthology.orchestra.run.smithi067.stderr: 31700K .......... .......... .......... .......... .......... 43% 153M 0s 2019-12-09T20:48:00.884 INFO:teuthology.orchestra.run.smithi067.stderr: 31750K .......... .......... .......... .......... .......... 43% 148M 0s 2019-12-09T20:48:00.884 INFO:teuthology.orchestra.run.smithi067.stderr: 31800K .......... .......... .......... .......... .......... 43% 159M 0s 2019-12-09T20:48:00.884 INFO:teuthology.orchestra.run.smithi067.stderr: 31850K .......... .......... .......... .......... .......... 43% 161M 0s 2019-12-09T20:48:00.884 INFO:teuthology.orchestra.run.smithi067.stderr: 31900K .......... .......... .......... .......... .......... 43% 156M 0s 2019-12-09T20:48:00.884 INFO:teuthology.orchestra.run.smithi067.stderr: 31950K .......... .......... .......... .......... .......... 43% 124M 0s 2019-12-09T20:48:00.884 INFO:teuthology.orchestra.run.smithi067.stderr: 32000K .......... .......... .......... .......... .......... 43% 149M 0s 2019-12-09T20:48:00.884 INFO:teuthology.orchestra.run.smithi067.stderr: 32050K .......... .......... .......... .......... .......... 43% 151M 0s 2019-12-09T20:48:00.885 INFO:teuthology.orchestra.run.smithi067.stderr: 32100K .......... .......... .......... .......... .......... 43% 148M 0s 2019-12-09T20:48:00.885 INFO:teuthology.orchestra.run.smithi067.stderr: 32150K .......... .......... .......... .......... .......... 43% 144M 0s 2019-12-09T20:48:00.885 INFO:teuthology.orchestra.run.smithi067.stderr: 32200K .......... .......... .......... .......... .......... 43% 147M 0s 2019-12-09T20:48:00.885 INFO:teuthology.orchestra.run.smithi067.stderr: 32250K .......... .......... .......... .......... .......... 43% 153M 0s 2019-12-09T20:48:00.885 INFO:teuthology.orchestra.run.smithi067.stderr: 32300K .......... .......... .......... .......... .......... 43% 151M 0s 2019-12-09T20:48:00.885 INFO:teuthology.orchestra.run.smithi067.stderr: 32350K .......... .......... .......... .......... .......... 44% 126M 0s 2019-12-09T20:48:00.885 INFO:teuthology.orchestra.run.smithi067.stderr: 32400K .......... .......... .......... .......... .......... 44% 158M 0s 2019-12-09T20:48:00.886 INFO:teuthology.orchestra.run.smithi067.stderr: 32450K .......... .......... .......... .......... .......... 44% 162M 0s 2019-12-09T20:48:00.886 INFO:teuthology.orchestra.run.smithi067.stderr: 32500K .......... .......... .......... .......... .......... 44% 152M 0s 2019-12-09T20:48:00.886 INFO:teuthology.orchestra.run.smithi067.stderr: 32550K .......... .......... .......... .......... .......... 44% 137M 0s 2019-12-09T20:48:00.886 INFO:teuthology.orchestra.run.smithi067.stderr: 32600K .......... .......... .......... .......... .......... 44% 156M 0s 2019-12-09T20:48:00.886 INFO:teuthology.orchestra.run.smithi067.stderr: 32650K .......... .......... .......... .......... .......... 44% 156M 0s 2019-12-09T20:48:00.886 INFO:teuthology.orchestra.run.smithi067.stderr: 32700K .......... .......... .......... .......... .......... 44% 155M 0s 2019-12-09T20:48:00.886 INFO:teuthology.orchestra.run.smithi067.stderr: 32750K .......... .......... .......... .......... .......... 44% 135M 0s 2019-12-09T20:48:00.887 INFO:teuthology.orchestra.run.smithi067.stderr: 32800K .......... .......... .......... .......... .......... 44% 152M 0s 2019-12-09T20:48:00.887 INFO:teuthology.orchestra.run.smithi067.stderr: 32850K .......... .......... .......... .......... .......... 44% 144M 0s 2019-12-09T20:48:00.887 INFO:teuthology.orchestra.run.smithi067.stderr: 32900K .......... .......... .......... .......... .......... 44% 153M 0s 2019-12-09T20:48:00.887 INFO:teuthology.orchestra.run.smithi067.stderr: 32950K .......... .......... .......... .......... .......... 44% 131M 0s 2019-12-09T20:48:00.887 INFO:teuthology.orchestra.run.smithi067.stderr: 33000K .......... .......... .......... .......... .......... 44% 155M 0s 2019-12-09T20:48:00.887 INFO:teuthology.orchestra.run.smithi067.stderr: 33050K .......... .......... .......... .......... .......... 44% 159M 0s 2019-12-09T20:48:00.887 INFO:teuthology.orchestra.run.smithi067.stderr: 33100K .......... .......... .......... .......... .......... 45% 167M 0s 2019-12-09T20:48:00.887 INFO:teuthology.orchestra.run.smithi067.stderr: 33150K .......... .......... .......... .......... .......... 45% 148M 0s 2019-12-09T20:48:00.888 INFO:teuthology.orchestra.run.smithi067.stderr: 33200K .......... .......... .......... .......... .......... 45% 157M 0s 2019-12-09T20:48:00.888 INFO:teuthology.orchestra.run.smithi067.stderr: 33250K .......... .......... .......... .......... .......... 45% 167M 0s 2019-12-09T20:48:00.888 INFO:teuthology.orchestra.run.smithi067.stderr: 33300K .......... .......... .......... .......... .......... 45% 156M 0s 2019-12-09T20:48:00.888 INFO:teuthology.orchestra.run.smithi067.stderr: 33350K .......... .......... .......... .......... .......... 45% 141M 0s 2019-12-09T20:48:00.888 INFO:teuthology.orchestra.run.smithi067.stderr: 33400K .......... .......... .......... .......... .......... 45% 158M 0s 2019-12-09T20:48:00.888 INFO:teuthology.orchestra.run.smithi067.stderr: 33450K .......... .......... .......... .......... .......... 45% 161M 0s 2019-12-09T20:48:00.888 INFO:teuthology.orchestra.run.smithi067.stderr: 33500K .......... .......... .......... .......... .......... 45% 163M 0s 2019-12-09T20:48:00.889 INFO:teuthology.orchestra.run.smithi067.stderr: 33550K .......... .......... .......... .......... .......... 45% 136M 0s 2019-12-09T20:48:00.889 INFO:teuthology.orchestra.run.smithi067.stderr: 33600K .......... .......... .......... .......... .......... 45% 160M 0s 2019-12-09T20:48:00.889 INFO:teuthology.orchestra.run.smithi067.stderr: 33650K .......... .......... .......... .......... .......... 45% 162M 0s 2019-12-09T20:48:00.889 INFO:teuthology.orchestra.run.smithi067.stderr: 33700K .......... .......... .......... .......... .......... 45% 168M 0s 2019-12-09T20:48:00.889 INFO:teuthology.orchestra.run.smithi067.stderr: 33750K .......... .......... .......... .......... .......... 45% 153M 0s 2019-12-09T20:48:00.889 INFO:teuthology.orchestra.run.smithi067.stderr: 33800K .......... .......... .......... .......... .......... 46% 154M 0s 2019-12-09T20:48:00.889 INFO:teuthology.orchestra.run.smithi067.stderr: 33850K .......... .......... .......... .......... .......... 46% 159M 0s 2019-12-09T20:48:00.890 INFO:teuthology.orchestra.run.smithi067.stderr: 33900K .......... .......... .......... .......... .......... 46% 156M 0s 2019-12-09T20:48:00.890 INFO:teuthology.orchestra.run.smithi067.stderr: 33950K .......... .......... .......... .......... .......... 46% 124M 0s 2019-12-09T20:48:00.890 INFO:teuthology.orchestra.run.smithi067.stderr: 34000K .......... .......... .......... .......... .......... 46% 166M 0s 2019-12-09T20:48:00.890 INFO:teuthology.orchestra.run.smithi067.stderr: 34050K .......... .......... .......... .......... .......... 46% 158M 0s 2019-12-09T20:48:00.890 INFO:teuthology.orchestra.run.smithi067.stderr: 34100K .......... .......... .......... .......... .......... 46% 155M 0s 2019-12-09T20:48:00.890 INFO:teuthology.orchestra.run.smithi067.stderr: 34150K .......... .......... .......... .......... .......... 46% 134M 0s 2019-12-09T20:48:00.890 INFO:teuthology.orchestra.run.smithi067.stderr: 34200K .......... .......... .......... .......... .......... 46% 165M 0s 2019-12-09T20:48:00.890 INFO:teuthology.orchestra.run.smithi067.stderr: 34250K .......... .......... .......... .......... .......... 46% 160M 0s 2019-12-09T20:48:00.891 INFO:teuthology.orchestra.run.smithi067.stderr: 34300K .......... .......... .......... .......... .......... 46% 163M 0s 2019-12-09T20:48:00.891 INFO:teuthology.orchestra.run.smithi067.stderr: 34350K .......... .......... .......... .......... .......... 46% 145M 0s 2019-12-09T20:48:00.891 INFO:teuthology.orchestra.run.smithi067.stderr: 34400K .......... .......... .......... .......... .......... 46% 169M 0s 2019-12-09T20:48:00.891 INFO:teuthology.orchestra.run.smithi067.stderr: 34450K .......... .......... .......... .......... .......... 46% 157M 0s 2019-12-09T20:48:00.891 INFO:teuthology.orchestra.run.smithi067.stderr: 34500K .......... .......... .......... .......... .......... 46% 157M 0s 2019-12-09T20:48:00.891 INFO:teuthology.orchestra.run.smithi067.stderr: 34550K .......... .......... .......... .......... .......... 47% 133M 0s 2019-12-09T20:48:00.891 INFO:teuthology.orchestra.run.smithi067.stderr: 34600K .......... .......... .......... .......... .......... 47% 164M 0s 2019-12-09T20:48:00.892 INFO:teuthology.orchestra.run.smithi067.stderr: 34650K .......... .......... .......... .......... .......... 47% 170M 0s 2019-12-09T20:48:00.892 INFO:teuthology.orchestra.run.smithi067.stderr: 34700K .......... .......... .......... .......... .......... 47% 174M 0s 2019-12-09T20:48:00.892 INFO:teuthology.orchestra.run.smithi067.stderr: 34750K .......... .......... .......... .......... .......... 47% 146M 0s 2019-12-09T20:48:00.892 INFO:teuthology.orchestra.run.smithi067.stderr: 34800K .......... .......... .......... .......... .......... 47% 180M 0s 2019-12-09T20:48:00.892 INFO:teuthology.orchestra.run.smithi067.stderr: 34850K .......... .......... .......... .......... .......... 47% 177M 0s 2019-12-09T20:48:00.892 INFO:teuthology.orchestra.run.smithi067.stderr: 34900K .......... .......... .......... .......... .......... 47% 183M 0s 2019-12-09T20:48:00.892 INFO:teuthology.orchestra.run.smithi067.stderr: 34950K .......... .......... .......... .......... .......... 47% 152M 0s 2019-12-09T20:48:00.893 INFO:teuthology.orchestra.run.smithi067.stderr: 35000K .......... .......... .......... .......... .......... 47% 173M 0s 2019-12-09T20:48:00.893 INFO:teuthology.orchestra.run.smithi067.stderr: 35050K .......... .......... .......... .......... .......... 47% 170M 0s 2019-12-09T20:48:00.893 INFO:teuthology.orchestra.run.smithi067.stderr: 35100K .......... .......... .......... .......... .......... 47% 160M 0s 2019-12-09T20:48:00.893 INFO:teuthology.orchestra.run.smithi067.stderr: 35150K .......... .......... .......... .......... .......... 47% 143M 0s 2019-12-09T20:48:00.893 INFO:teuthology.orchestra.run.smithi067.stderr: 35200K .......... .......... .......... .......... .......... 47% 176M 0s 2019-12-09T20:48:00.893 INFO:teuthology.orchestra.run.smithi067.stderr: 35250K .......... .......... .......... .......... .......... 47% 166M 0s 2019-12-09T20:48:00.893 INFO:teuthology.orchestra.run.smithi067.stderr: 35300K .......... .......... .......... .......... .......... 48% 169M 0s 2019-12-09T20:48:00.893 INFO:teuthology.orchestra.run.smithi067.stderr: 35350K .......... .......... .......... .......... .......... 48% 154M 0s 2019-12-09T20:48:00.894 INFO:teuthology.orchestra.run.smithi067.stderr: 35400K .......... .......... .......... .......... .......... 48% 168M 0s 2019-12-09T20:48:00.894 INFO:teuthology.orchestra.run.smithi067.stderr: 35450K .......... .......... .......... .......... .......... 48% 158M 0s 2019-12-09T20:48:00.894 INFO:teuthology.orchestra.run.smithi067.stderr: 35500K .......... .......... .......... .......... .......... 48% 167M 0s 2019-12-09T20:48:00.894 INFO:teuthology.orchestra.run.smithi067.stderr: 35550K .......... .......... .......... .......... .......... 48% 147M 0s 2019-12-09T20:48:00.894 INFO:teuthology.orchestra.run.smithi067.stderr: 35600K .......... .......... .......... .......... .......... 48% 181M 0s 2019-12-09T20:48:00.894 INFO:teuthology.orchestra.run.smithi067.stderr: 35650K .......... .......... .......... .......... .......... 48% 182M 0s 2019-12-09T20:48:00.894 INFO:teuthology.orchestra.run.smithi067.stderr: 35700K .......... .......... .......... .......... .......... 48% 174M 0s 2019-12-09T20:48:00.895 INFO:teuthology.orchestra.run.smithi067.stderr: 35750K .......... .......... .......... .......... .......... 48% 141M 0s 2019-12-09T20:48:00.895 INFO:teuthology.orchestra.run.smithi067.stderr: 35800K .......... .......... .......... .......... .......... 48% 170M 0s 2019-12-09T20:48:00.895 INFO:teuthology.orchestra.run.smithi067.stderr: 35850K .......... .......... .......... .......... .......... 48% 179M 0s 2019-12-09T20:48:00.895 INFO:teuthology.orchestra.run.smithi067.stderr: 35900K .......... .......... .......... .......... .......... 48% 173M 0s 2019-12-09T20:48:00.895 INFO:teuthology.orchestra.run.smithi067.stderr: 35950K .......... .......... .......... .......... .......... 48% 154M 0s 2019-12-09T20:48:00.895 INFO:teuthology.orchestra.run.smithi067.stderr: 36000K .......... .......... .......... .......... .......... 49% 163M 0s 2019-12-09T20:48:00.895 INFO:teuthology.orchestra.run.smithi067.stderr: 36050K .......... .......... .......... .......... .......... 49% 153M 0s 2019-12-09T20:48:00.896 INFO:teuthology.orchestra.run.smithi067.stderr: 36100K .......... .......... .......... .......... .......... 49% 171M 0s 2019-12-09T20:48:00.896 INFO:teuthology.orchestra.run.smithi067.stderr: 36150K .......... .......... .......... .......... .......... 49% 155M 0s 2019-12-09T20:48:00.896 INFO:teuthology.orchestra.run.smithi067.stderr: 36200K .......... .......... .......... .......... .......... 49% 172M 0s 2019-12-09T20:48:00.896 INFO:teuthology.orchestra.run.smithi067.stderr: 36250K .......... .......... .......... .......... .......... 49% 175M 0s 2019-12-09T20:48:00.896 INFO:teuthology.orchestra.run.smithi067.stderr: 36300K .......... .......... .......... .......... .......... 49% 173M 0s 2019-12-09T20:48:00.896 INFO:teuthology.orchestra.run.smithi067.stderr: 36350K .......... .......... .......... .......... .......... 49% 152M 0s 2019-12-09T20:48:00.896 INFO:teuthology.orchestra.run.smithi067.stderr: 36400K .......... .......... .......... .......... .......... 49% 181M 0s 2019-12-09T20:48:00.897 INFO:teuthology.orchestra.run.smithi067.stderr: 36450K .......... .......... .......... .......... .......... 49% 180M 0s 2019-12-09T20:48:00.897 INFO:teuthology.orchestra.run.smithi067.stderr: 36500K .......... .......... .......... .......... .......... 49% 182M 0s 2019-12-09T20:48:00.897 INFO:teuthology.orchestra.run.smithi067.stderr: 36550K .......... .......... .......... .......... .......... 49% 168M 0s 2019-12-09T20:48:00.897 INFO:teuthology.orchestra.run.smithi067.stderr: 36600K .......... .......... .......... .......... .......... 49% 184M 0s 2019-12-09T20:48:00.897 INFO:teuthology.orchestra.run.smithi067.stderr: 36650K .......... .......... .......... .......... .......... 49% 193M 0s 2019-12-09T20:48:00.897 INFO:teuthology.orchestra.run.smithi067.stderr: 36700K .......... .......... .......... .......... .......... 49% 193M 0s 2019-12-09T20:48:00.897 INFO:teuthology.orchestra.run.smithi067.stderr: 36750K .......... .......... .......... .......... .......... 50% 147M 0s 2019-12-09T20:48:00.897 INFO:teuthology.orchestra.run.smithi067.stderr: 36800K .......... .......... .......... .......... .......... 50% 189M 0s 2019-12-09T20:48:00.898 INFO:teuthology.orchestra.run.smithi067.stderr: 36850K .......... .......... .......... .......... .......... 50% 187M 0s 2019-12-09T20:48:00.898 INFO:teuthology.orchestra.run.smithi067.stderr: 36900K .......... .......... .......... .......... .......... 50% 189M 0s 2019-12-09T20:48:00.898 INFO:teuthology.orchestra.run.smithi067.stderr: 36950K .......... .......... .......... .......... .......... 50% 170M 0s 2019-12-09T20:48:00.898 INFO:teuthology.orchestra.run.smithi067.stderr: 37000K .......... .......... .......... .......... .......... 50% 187M 0s 2019-12-09T20:48:00.898 INFO:teuthology.orchestra.run.smithi067.stderr: 37050K .......... .......... .......... .......... .......... 50% 198M 0s 2019-12-09T20:48:00.898 INFO:teuthology.orchestra.run.smithi067.stderr: 37100K .......... .......... .......... .......... .......... 50% 189M 0s 2019-12-09T20:48:00.898 INFO:teuthology.orchestra.run.smithi067.stderr: 37150K .......... .......... .......... .......... .......... 50% 158M 0s 2019-12-09T20:48:00.899 INFO:teuthology.orchestra.run.smithi067.stderr: 37200K .......... .......... .......... .......... .......... 50% 185M 0s 2019-12-09T20:48:00.899 INFO:teuthology.orchestra.run.smithi067.stderr: 37250K .......... .......... .......... .......... .......... 50% 179M 0s 2019-12-09T20:48:00.899 INFO:teuthology.orchestra.run.smithi067.stderr: 37300K .......... .......... .......... .......... .......... 50% 184M 0s 2019-12-09T20:48:00.899 INFO:teuthology.orchestra.run.smithi067.stderr: 37350K .......... .......... .......... .......... .......... 50% 154M 0s 2019-12-09T20:48:00.899 INFO:teuthology.orchestra.run.smithi067.stderr: 37400K .......... .......... .......... .......... .......... 50% 187M 0s 2019-12-09T20:48:00.899 INFO:teuthology.orchestra.run.smithi067.stderr: 37450K .......... .......... .......... .......... .......... 50% 180M 0s 2019-12-09T20:48:00.899 INFO:teuthology.orchestra.run.smithi067.stderr: 37500K .......... .......... .......... .......... .......... 51% 186M 0s 2019-12-09T20:48:00.899 INFO:teuthology.orchestra.run.smithi067.stderr: 37550K .......... .......... .......... .......... .......... 51% 165M 0s 2019-12-09T20:48:00.900 INFO:teuthology.orchestra.run.smithi067.stderr: 37600K .......... .......... .......... .......... .......... 51% 181M 0s 2019-12-09T20:48:00.900 INFO:teuthology.orchestra.run.smithi067.stderr: 37650K .......... .......... .......... .......... .......... 51% 175M 0s 2019-12-09T20:48:00.900 INFO:teuthology.orchestra.run.smithi067.stderr: 37700K .......... .......... .......... .......... .......... 51% 175M 0s 2019-12-09T20:48:00.900 INFO:teuthology.orchestra.run.smithi067.stderr: 37750K .......... .......... .......... .......... .......... 51% 150M 0s 2019-12-09T20:48:00.900 INFO:teuthology.orchestra.run.smithi067.stderr: 37800K .......... .......... .......... .......... .......... 51% 179M 0s 2019-12-09T20:48:00.900 INFO:teuthology.orchestra.run.smithi067.stderr: 37850K .......... .......... .......... .......... .......... 51% 176M 0s 2019-12-09T20:48:00.900 INFO:teuthology.orchestra.run.smithi067.stderr: 37900K .......... .......... .......... .......... .......... 51% 190M 0s 2019-12-09T20:48:00.901 INFO:teuthology.orchestra.run.smithi067.stderr: 37950K .......... .......... .......... .......... .......... 51% 163M 0s 2019-12-09T20:48:00.901 INFO:teuthology.orchestra.run.smithi067.stderr: 38000K .......... .......... .......... .......... .......... 51% 179M 0s 2019-12-09T20:48:00.901 INFO:teuthology.orchestra.run.smithi067.stderr: 38050K .......... .......... .......... .......... .......... 51% 178M 0s 2019-12-09T20:48:00.901 INFO:teuthology.orchestra.run.smithi067.stderr: 38100K .......... .......... .......... .......... .......... 51% 178M 0s 2019-12-09T20:48:00.901 INFO:teuthology.orchestra.run.smithi067.stderr: 38150K .......... .......... .......... .......... .......... 51% 152M 0s 2019-12-09T20:48:00.901 INFO:teuthology.orchestra.run.smithi067.stderr: 38200K .......... .......... .......... .......... .......... 51% 194M 0s 2019-12-09T20:48:00.901 INFO:teuthology.orchestra.run.smithi067.stderr: 38250K .......... .......... .......... .......... .......... 52% 198M 0s 2019-12-09T20:48:00.902 INFO:teuthology.orchestra.run.smithi067.stderr: 38300K .......... .......... .......... .......... .......... 52% 198M 0s 2019-12-09T20:48:00.902 INFO:teuthology.orchestra.run.smithi067.stderr: 38350K .......... .......... .......... .......... .......... 52% 169M 0s 2019-12-09T20:48:00.902 INFO:teuthology.orchestra.run.smithi067.stderr: 38400K .......... .......... .......... .......... .......... 52% 195M 0s 2019-12-09T20:48:00.902 INFO:teuthology.orchestra.run.smithi067.stderr: 38450K .......... .......... .......... .......... .......... 52% 183M 0s 2019-12-09T20:48:00.902 INFO:teuthology.orchestra.run.smithi067.stderr: 38500K .......... .......... .......... .......... .......... 52% 191M 0s 2019-12-09T20:48:00.902 INFO:teuthology.orchestra.run.smithi067.stderr: 38550K .......... .......... .......... .......... .......... 52% 181M 0s 2019-12-09T20:48:00.902 INFO:teuthology.orchestra.run.smithi067.stderr: 38600K .......... .......... .......... .......... .......... 52% 195M 0s 2019-12-09T20:48:00.903 INFO:teuthology.orchestra.run.smithi067.stderr: 38650K .......... .......... .......... .......... .......... 52% 179M 0s 2019-12-09T20:48:00.903 INFO:teuthology.orchestra.run.smithi067.stderr: 38700K .......... .......... .......... .......... .......... 52% 201M 0s 2019-12-09T20:48:00.903 INFO:teuthology.orchestra.run.smithi067.stderr: 38750K .......... .......... .......... .......... .......... 52% 181M 0s 2019-12-09T20:48:00.903 INFO:teuthology.orchestra.run.smithi067.stderr: 38800K .......... .......... .......... .......... .......... 52% 199M 0s 2019-12-09T20:48:00.903 INFO:teuthology.orchestra.run.smithi067.stderr: 38850K .......... .......... .......... .......... .......... 52% 199M 0s 2019-12-09T20:48:00.903 INFO:teuthology.orchestra.run.smithi067.stderr: 38900K .......... .......... .......... .......... .......... 52% 192M 0s 2019-12-09T20:48:00.903 INFO:teuthology.orchestra.run.smithi067.stderr: 38950K .......... .......... .......... .......... .......... 53% 154M 0s 2019-12-09T20:48:00.904 INFO:teuthology.orchestra.run.smithi067.stderr: 39000K .......... .......... .......... .......... .......... 53% 198M 0s 2019-12-09T20:48:00.904 INFO:teuthology.orchestra.run.smithi067.stderr: 39050K .......... .......... .......... .......... .......... 53% 182M 0s 2019-12-09T20:48:00.904 INFO:teuthology.orchestra.run.smithi067.stderr: 39100K .......... .......... .......... .......... .......... 53% 192M 0s 2019-12-09T20:48:00.904 INFO:teuthology.orchestra.run.smithi067.stderr: 39150K .......... .......... .......... .......... .......... 53% 173M 0s 2019-12-09T20:48:00.904 INFO:teuthology.orchestra.run.smithi067.stderr: 39200K .......... .......... .......... .......... .......... 53% 182M 0s 2019-12-09T20:48:00.904 INFO:teuthology.orchestra.run.smithi067.stderr: 39250K .......... .......... .......... .......... .......... 53% 193M 0s 2019-12-09T20:48:00.904 INFO:teuthology.orchestra.run.smithi067.stderr: 39300K .......... .......... .......... .......... .......... 53% 192M 0s 2019-12-09T20:48:00.904 INFO:teuthology.orchestra.run.smithi067.stderr: 39350K .......... .......... .......... .......... .......... 53% 163M 0s 2019-12-09T20:48:00.905 INFO:teuthology.orchestra.run.smithi067.stderr: 39400K .......... .......... .......... .......... .......... 53% 195M 0s 2019-12-09T20:48:00.905 INFO:teuthology.orchestra.run.smithi067.stderr: 39450K .......... .......... .......... .......... .......... 53% 180M 0s 2019-12-09T20:48:00.905 INFO:teuthology.orchestra.run.smithi067.stderr: 39500K .......... .......... .......... .......... .......... 53% 204M 0s 2019-12-09T20:48:00.905 INFO:teuthology.orchestra.run.smithi067.stderr: 39550K .......... .......... .......... .......... .......... 53% 157M 0s 2019-12-09T20:48:00.905 INFO:teuthology.orchestra.run.smithi067.stderr: 39600K .......... .......... .......... .......... .......... 53% 190M 0s 2019-12-09T20:48:00.905 INFO:teuthology.orchestra.run.smithi067.stderr: 39650K .......... .......... .......... .......... .......... 53% 200M 0s 2019-12-09T20:48:00.905 INFO:teuthology.orchestra.run.smithi067.stderr: 39700K .......... .......... .......... .......... .......... 54% 190M 0s 2019-12-09T20:48:00.906 INFO:teuthology.orchestra.run.smithi067.stderr: 39750K .......... .......... .......... .......... .......... 54% 164M 0s 2019-12-09T20:48:00.906 INFO:teuthology.orchestra.run.smithi067.stderr: 39800K .......... .......... .......... .......... .......... 54% 193M 0s 2019-12-09T20:48:00.906 INFO:teuthology.orchestra.run.smithi067.stderr: 39850K .......... .......... .......... .......... .......... 54% 176M 0s 2019-12-09T20:48:00.906 INFO:teuthology.orchestra.run.smithi067.stderr: 39900K .......... .......... .......... .......... .......... 54% 193M 0s 2019-12-09T20:48:00.906 INFO:teuthology.orchestra.run.smithi067.stderr: 39950K .......... .......... .......... .......... .......... 54% 149M 0s 2019-12-09T20:48:00.906 INFO:teuthology.orchestra.run.smithi067.stderr: 40000K .......... .......... .......... .......... .......... 54% 194M 0s 2019-12-09T20:48:00.906 INFO:teuthology.orchestra.run.smithi067.stderr: 40050K .......... .......... .......... .......... .......... 54% 204M 0s 2019-12-09T20:48:00.907 INFO:teuthology.orchestra.run.smithi067.stderr: 40100K .......... .......... .......... .......... .......... 54% 208M 0s 2019-12-09T20:48:00.907 INFO:teuthology.orchestra.run.smithi067.stderr: 40150K .......... .......... .......... .......... .......... 54% 173M 0s 2019-12-09T20:48:00.907 INFO:teuthology.orchestra.run.smithi067.stderr: 40200K .......... .......... .......... .......... .......... 54% 195M 0s 2019-12-09T20:48:00.907 INFO:teuthology.orchestra.run.smithi067.stderr: 40250K .......... .......... .......... .......... .......... 54% 211M 0s 2019-12-09T20:48:00.907 INFO:teuthology.orchestra.run.smithi067.stderr: 40300K .......... .......... .......... .......... .......... 54% 207M 0s 2019-12-09T20:48:00.907 INFO:teuthology.orchestra.run.smithi067.stderr: 40350K .......... .......... .......... .......... .......... 54% 173M 0s 2019-12-09T20:48:00.907 INFO:teuthology.orchestra.run.smithi067.stderr: 40400K .......... .......... .......... .......... .......... 54% 210M 0s 2019-12-09T20:48:00.907 INFO:teuthology.orchestra.run.smithi067.stderr: 40450K .......... .......... .......... .......... .......... 55% 191M 0s 2019-12-09T20:48:00.908 INFO:teuthology.orchestra.run.smithi067.stderr: 40500K .......... .......... .......... .......... .......... 55% 205M 0s 2019-12-09T20:48:00.908 INFO:teuthology.orchestra.run.smithi067.stderr: 40550K .......... .......... .......... .......... .......... 55% 174M 0s 2019-12-09T20:48:00.908 INFO:teuthology.orchestra.run.smithi067.stderr: 40600K .......... .......... .......... .......... .......... 55% 183M 0s 2019-12-09T20:48:00.908 INFO:teuthology.orchestra.run.smithi067.stderr: 40650K .......... .......... .......... .......... .......... 55% 188M 0s 2019-12-09T20:48:00.908 INFO:teuthology.orchestra.run.smithi067.stderr: 40700K .......... .......... .......... .......... .......... 55% 200M 0s 2019-12-09T20:48:00.908 INFO:teuthology.orchestra.run.smithi067.stderr: 40750K .......... .......... .......... .......... .......... 55% 159M 0s 2019-12-09T20:48:00.908 INFO:teuthology.orchestra.run.smithi067.stderr: 40800K .......... .......... .......... .......... .......... 55% 200M 0s 2019-12-09T20:48:00.909 INFO:teuthology.orchestra.run.smithi067.stderr: 40850K .......... .......... .......... .......... .......... 55% 181M 0s 2019-12-09T20:48:00.909 INFO:teuthology.orchestra.run.smithi067.stderr: 40900K .......... .......... .......... .......... .......... 55% 203M 0s 2019-12-09T20:48:00.909 INFO:teuthology.orchestra.run.smithi067.stderr: 40950K .......... .......... .......... .......... .......... 55% 161M 0s 2019-12-09T20:48:00.909 INFO:teuthology.orchestra.run.smithi067.stderr: 41000K .......... .......... .......... .......... .......... 55% 202M 0s 2019-12-09T20:48:00.909 INFO:teuthology.orchestra.run.smithi067.stderr: 41050K .......... .......... .......... .......... .......... 55% 188M 0s 2019-12-09T20:48:00.909 INFO:teuthology.orchestra.run.smithi067.stderr: 41100K .......... .......... .......... .......... .......... 55% 196M 0s 2019-12-09T20:48:00.909 INFO:teuthology.orchestra.run.smithi067.stderr: 41150K .......... .......... .......... .......... .......... 56% 175M 0s 2019-12-09T20:48:00.910 INFO:teuthology.orchestra.run.smithi067.stderr: 41200K .......... .......... .......... .......... .......... 56% 188M 0s 2019-12-09T20:48:00.910 INFO:teuthology.orchestra.run.smithi067.stderr: 41250K .......... .......... .......... .......... .......... 56% 181M 0s 2019-12-09T20:48:00.910 INFO:teuthology.orchestra.run.smithi067.stderr: 41300K .......... .......... .......... .......... .......... 56% 164M 0s 2019-12-09T20:48:00.910 INFO:teuthology.orchestra.run.smithi067.stderr: 41350K .......... .......... .......... .......... .......... 56% 170M 0s 2019-12-09T20:48:00.910 INFO:teuthology.orchestra.run.smithi067.stderr: 41400K .......... .......... .......... .......... .......... 56% 184M 0s 2019-12-09T20:48:00.910 INFO:teuthology.orchestra.run.smithi067.stderr: 41450K .......... .......... .......... .......... .......... 56% 195M 0s 2019-12-09T20:48:00.910 INFO:teuthology.orchestra.run.smithi067.stderr: 41500K .......... .......... .......... .......... .......... 56% 182M 0s 2019-12-09T20:48:00.911 INFO:teuthology.orchestra.run.smithi067.stderr: 41550K .......... .......... .......... .......... .......... 56% 155M 0s 2019-12-09T20:48:00.911 INFO:teuthology.orchestra.run.smithi067.stderr: 41600K .......... .......... .......... .......... .......... 56% 191M 0s 2019-12-09T20:48:00.911 INFO:teuthology.orchestra.run.smithi067.stderr: 41650K .......... .......... .......... .......... .......... 56% 187M 0s 2019-12-09T20:48:00.911 INFO:teuthology.orchestra.run.smithi067.stderr: 41700K .......... .......... .......... .......... .......... 56% 189M 0s 2019-12-09T20:48:00.911 INFO:teuthology.orchestra.run.smithi067.stderr: 41750K .......... .......... .......... .......... .......... 56% 176M 0s 2019-12-09T20:48:00.911 INFO:teuthology.orchestra.run.smithi067.stderr: 41800K .......... .......... .......... .......... .......... 56% 191M 0s 2019-12-09T20:48:00.911 INFO:teuthology.orchestra.run.smithi067.stderr: 41850K .......... .......... .......... .......... .......... 56% 185M 0s 2019-12-09T20:48:00.912 INFO:teuthology.orchestra.run.smithi067.stderr: 41900K .......... .......... .......... .......... .......... 57% 199M 0s 2019-12-09T20:48:00.912 INFO:teuthology.orchestra.run.smithi067.stderr: 41950K .......... .......... .......... .......... .......... 57% 180M 0s 2019-12-09T20:48:00.912 INFO:teuthology.orchestra.run.smithi067.stderr: 42000K .......... .......... .......... .......... .......... 57% 205M 0s 2019-12-09T20:48:00.912 INFO:teuthology.orchestra.run.smithi067.stderr: 42050K .......... .......... .......... .......... .......... 57% 201M 0s 2019-12-09T20:48:00.912 INFO:teuthology.orchestra.run.smithi067.stderr: 42100K .......... .......... .......... .......... .......... 57% 206M 0s 2019-12-09T20:48:00.912 INFO:teuthology.orchestra.run.smithi067.stderr: 42150K .......... .......... .......... .......... .......... 57% 172M 0s 2019-12-09T20:48:00.912 INFO:teuthology.orchestra.run.smithi067.stderr: 42200K .......... .......... .......... .......... .......... 57% 206M 0s 2019-12-09T20:48:00.913 INFO:teuthology.orchestra.run.smithi067.stderr: 42250K .......... .......... .......... .......... .......... 57% 190M 0s 2019-12-09T20:48:00.913 INFO:teuthology.orchestra.run.smithi067.stderr: 42300K .......... .......... .......... .......... .......... 57% 197M 0s 2019-12-09T20:48:00.913 INFO:teuthology.orchestra.run.smithi067.stderr: 42350K .......... .......... .......... .......... .......... 57% 171M 0s 2019-12-09T20:48:00.913 INFO:teuthology.orchestra.run.smithi067.stderr: 42400K .......... .......... .......... .......... .......... 57% 198M 0s 2019-12-09T20:48:00.913 INFO:teuthology.orchestra.run.smithi067.stderr: 42450K .......... .......... .......... .......... .......... 57% 188M 0s 2019-12-09T20:48:00.913 INFO:teuthology.orchestra.run.smithi067.stderr: 42500K .......... .......... .......... .......... .......... 57% 192M 0s 2019-12-09T20:48:00.913 INFO:teuthology.orchestra.run.smithi067.stderr: 42550K .......... .......... .......... .......... .......... 57% 167M 0s 2019-12-09T20:48:00.913 INFO:teuthology.orchestra.run.smithi067.stderr: 42600K .......... .......... .......... .......... .......... 57% 211M 0s 2019-12-09T20:48:00.914 INFO:teuthology.orchestra.run.smithi067.stderr: 42650K .......... .......... .......... .......... .......... 58% 202M 0s 2019-12-09T20:48:00.914 INFO:teuthology.orchestra.run.smithi067.stderr: 42700K .......... .......... .......... .......... .......... 58% 220M 0s 2019-12-09T20:48:00.914 INFO:teuthology.orchestra.run.smithi067.stderr: 42750K .......... .......... .......... .......... .......... 58% 175M 0s 2019-12-09T20:48:00.914 INFO:teuthology.orchestra.run.smithi067.stderr: 42800K .......... .......... .......... .......... .......... 58% 202M 0s 2019-12-09T20:48:00.914 INFO:teuthology.orchestra.run.smithi067.stderr: 42850K .......... .......... .......... .......... .......... 58% 197M 0s 2019-12-09T20:48:00.914 INFO:teuthology.orchestra.run.smithi067.stderr: 42900K .......... .......... .......... .......... .......... 58% 205M 0s 2019-12-09T20:48:00.914 INFO:teuthology.orchestra.run.smithi067.stderr: 42950K .......... .......... .......... .......... .......... 58% 191M 0s 2019-12-09T20:48:00.915 INFO:teuthology.orchestra.run.smithi067.stderr: 43000K .......... .......... .......... .......... .......... 58% 205M 0s 2019-12-09T20:48:00.915 INFO:teuthology.orchestra.run.smithi067.stderr: 43050K .......... .......... .......... .......... .......... 58% 202M 0s 2019-12-09T20:48:00.915 INFO:teuthology.orchestra.run.smithi067.stderr: 43100K .......... .......... .......... .......... .......... 58% 202M 0s 2019-12-09T20:48:00.915 INFO:teuthology.orchestra.run.smithi067.stderr: 43150K .......... .......... .......... .......... .......... 58% 173M 0s 2019-12-09T20:48:00.915 INFO:teuthology.orchestra.run.smithi067.stderr: 43200K .......... .......... .......... .......... .......... 58% 209M 0s 2019-12-09T20:48:00.915 INFO:teuthology.orchestra.run.smithi067.stderr: 43250K .......... .......... .......... .......... .......... 58% 201M 0s 2019-12-09T20:48:00.915 INFO:teuthology.orchestra.run.smithi067.stderr: 43300K .......... .......... .......... .......... .......... 58% 220M 0s 2019-12-09T20:48:00.916 INFO:teuthology.orchestra.run.smithi067.stderr: 43350K .......... .......... .......... .......... .......... 59% 177M 0s 2019-12-09T20:48:00.916 INFO:teuthology.orchestra.run.smithi067.stderr: 43400K .......... .......... .......... .......... .......... 59% 217M 0s 2019-12-09T20:48:00.916 INFO:teuthology.orchestra.run.smithi067.stderr: 43450K .......... .......... .......... .......... .......... 59% 201M 0s 2019-12-09T20:48:00.916 INFO:teuthology.orchestra.run.smithi067.stderr: 43500K .......... .......... .......... .......... .......... 59% 201M 0s 2019-12-09T20:48:00.916 INFO:teuthology.orchestra.run.smithi067.stderr: 43550K .......... .......... .......... .......... .......... 59% 195M 0s 2019-12-09T20:48:00.916 INFO:teuthology.orchestra.run.smithi067.stderr: 43600K .......... .......... .......... .......... .......... 59% 218M 0s 2019-12-09T20:48:00.916 INFO:teuthology.orchestra.run.smithi067.stderr: 43650K .......... .......... .......... .......... .......... 59% 220M 0s 2019-12-09T20:48:00.917 INFO:teuthology.orchestra.run.smithi067.stderr: 43700K .......... .......... .......... .......... .......... 59% 203M 0s 2019-12-09T20:48:00.917 INFO:teuthology.orchestra.run.smithi067.stderr: 43750K .......... .......... .......... .......... .......... 59% 188M 0s 2019-12-09T20:48:00.917 INFO:teuthology.orchestra.run.smithi067.stderr: 43800K .......... .......... .......... .......... .......... 59% 218M 0s 2019-12-09T20:48:00.917 INFO:teuthology.orchestra.run.smithi067.stderr: 43850K .......... .......... .......... .......... .......... 59% 225M 0s 2019-12-09T20:48:00.917 INFO:teuthology.orchestra.run.smithi067.stderr: 43900K .......... .......... .......... .......... .......... 59% 198M 0s 2019-12-09T20:48:00.917 INFO:teuthology.orchestra.run.smithi067.stderr: 43950K .......... .......... .......... .......... .......... 59% 190M 0s 2019-12-09T20:48:00.917 INFO:teuthology.orchestra.run.smithi067.stderr: 44000K .......... .......... .......... .......... .......... 59% 211M 0s 2019-12-09T20:48:00.918 INFO:teuthology.orchestra.run.smithi067.stderr: 44050K .......... .......... .......... .......... .......... 59% 210M 0s 2019-12-09T20:48:00.918 INFO:teuthology.orchestra.run.smithi067.stderr: 44100K .......... .......... .......... .......... .......... 60% 218M 0s 2019-12-09T20:48:00.918 INFO:teuthology.orchestra.run.smithi067.stderr: 44150K .......... .......... .......... .......... .......... 60% 204M 0s 2019-12-09T20:48:00.918 INFO:teuthology.orchestra.run.smithi067.stderr: 44200K .......... .......... .......... .......... .......... 60% 225M 0s 2019-12-09T20:48:00.918 INFO:teuthology.orchestra.run.smithi067.stderr: 44250K .......... .......... .......... .......... .......... 60% 224M 0s 2019-12-09T20:48:00.918 INFO:teuthology.orchestra.run.smithi067.stderr: 44300K .......... .......... .......... .......... .......... 60% 218M 0s 2019-12-09T20:48:00.918 INFO:teuthology.orchestra.run.smithi067.stderr: 44350K .......... .......... .......... .......... .......... 60% 108M 0s 2019-12-09T20:48:00.918 INFO:teuthology.orchestra.run.smithi067.stderr: 44400K .......... .......... .......... .......... .......... 60% 277M 0s 2019-12-09T20:48:00.919 INFO:teuthology.orchestra.run.smithi067.stderr: 44450K .......... .......... .......... .......... .......... 60% 351M 0s 2019-12-09T20:48:00.919 INFO:teuthology.orchestra.run.smithi067.stderr: 44500K .......... .......... .......... .......... .......... 60% 60.5M 0s 2019-12-09T20:48:00.919 INFO:teuthology.orchestra.run.smithi067.stderr: 44550K .......... .......... .......... .......... .......... 60% 216M 0s 2019-12-09T20:48:00.919 INFO:teuthology.orchestra.run.smithi067.stderr: 44600K .......... .......... .......... .......... .......... 60% 393M 0s 2019-12-09T20:48:00.919 INFO:teuthology.orchestra.run.smithi067.stderr: 44650K .......... .......... .......... .......... .......... 60% 85.7M 0s 2019-12-09T20:48:00.919 INFO:teuthology.orchestra.run.smithi067.stderr: 44700K .......... .......... .......... .......... .......... 60% 307M 0s 2019-12-09T20:48:00.919 INFO:teuthology.orchestra.run.smithi067.stderr: 44750K .......... .......... .......... .......... .......... 60% 350M 0s 2019-12-09T20:48:00.920 INFO:teuthology.orchestra.run.smithi067.stderr: 44800K .......... .......... .......... .......... .......... 60% 81.5M 0s 2019-12-09T20:48:00.920 INFO:teuthology.orchestra.run.smithi067.stderr: 44850K .......... .......... .......... .......... .......... 61% 311M 0s 2019-12-09T20:48:00.920 INFO:teuthology.orchestra.run.smithi067.stderr: 44900K .......... .......... .......... .......... .......... 61% 370M 0s 2019-12-09T20:48:00.920 INFO:teuthology.orchestra.run.smithi067.stderr: 44950K .......... .......... .......... .......... .......... 61% 313M 0s 2019-12-09T20:48:00.920 INFO:teuthology.orchestra.run.smithi067.stderr: 45000K .......... .......... .......... .......... .......... 61% 371M 0s 2019-12-09T20:48:00.920 INFO:teuthology.orchestra.run.smithi067.stderr: 45050K .......... .......... .......... .......... .......... 61% 34.7M 0s 2019-12-09T20:48:00.920 INFO:teuthology.orchestra.run.smithi067.stderr: 45100K .......... .......... .......... .......... .......... 61% 276M 0s 2019-12-09T20:48:00.921 INFO:teuthology.orchestra.run.smithi067.stderr: 45150K .......... .......... .......... .......... .......... 61% 330M 0s 2019-12-09T20:48:00.921 INFO:teuthology.orchestra.run.smithi067.stderr: 45200K .......... .......... .......... .......... .......... 61% 153M 0s 2019-12-09T20:48:00.921 INFO:teuthology.orchestra.run.smithi067.stderr: 45250K .......... .......... .......... .......... .......... 61% 373M 0s 2019-12-09T20:48:00.921 INFO:teuthology.orchestra.run.smithi067.stderr: 45300K .......... .......... .......... .......... .......... 61% 387M 0s 2019-12-09T20:48:00.921 INFO:teuthology.orchestra.run.smithi067.stderr: 45350K .......... .......... .......... .......... .......... 61% 325M 0s 2019-12-09T20:48:00.921 INFO:teuthology.orchestra.run.smithi067.stderr: 45400K .......... .......... .......... .......... .......... 61% 355M 0s 2019-12-09T20:48:00.921 INFO:teuthology.orchestra.run.smithi067.stderr: 45450K .......... .......... .......... .......... .......... 61% 342M 0s 2019-12-09T20:48:00.922 INFO:teuthology.orchestra.run.smithi067.stderr: 45500K .......... .......... .......... .......... .......... 61% 167M 0s 2019-12-09T20:48:00.922 INFO:teuthology.orchestra.run.smithi067.stderr: 45550K .......... .......... .......... .......... .......... 61% 227M 0s 2019-12-09T20:48:00.922 INFO:teuthology.orchestra.run.smithi067.stderr: 45600K .......... .......... .......... .......... .......... 62% 295M 0s 2019-12-09T20:48:00.922 INFO:teuthology.orchestra.run.smithi067.stderr: 45650K .......... .......... .......... .......... .......... 62% 357M 0s 2019-12-09T20:48:00.922 INFO:teuthology.orchestra.run.smithi067.stderr: 45700K .......... .......... .......... .......... .......... 62% 53.0M 0s 2019-12-09T20:48:00.922 INFO:teuthology.orchestra.run.smithi067.stderr: 45750K .......... .......... .......... .......... .......... 62% 240M 0s 2019-12-09T20:48:00.922 INFO:teuthology.orchestra.run.smithi067.stderr: 45800K .......... .......... .......... .......... .......... 62% 361M 0s 2019-12-09T20:48:00.922 INFO:teuthology.orchestra.run.smithi067.stderr: 45850K .......... .......... .......... .......... .......... 62% 351M 0s 2019-12-09T20:48:00.923 INFO:teuthology.orchestra.run.smithi067.stderr: 45900K .......... .......... .......... .......... .......... 62% 358M 0s 2019-12-09T20:48:00.923 INFO:teuthology.orchestra.run.smithi067.stderr: 45950K .......... .......... .......... .......... .......... 62% 344M 0s 2019-12-09T20:48:00.923 INFO:teuthology.orchestra.run.smithi067.stderr: 46000K .......... .......... .......... .......... .......... 62% 27.9M 0s 2019-12-09T20:48:00.923 INFO:teuthology.orchestra.run.smithi067.stderr: 46050K .......... .......... .......... .......... .......... 62% 335M 0s 2019-12-09T20:48:00.923 INFO:teuthology.orchestra.run.smithi067.stderr: 46100K .......... .......... .......... .......... .......... 62% 401M 0s 2019-12-09T20:48:00.923 INFO:teuthology.orchestra.run.smithi067.stderr: 46150K .......... .......... .......... .......... .......... 62% 156M 0s 2019-12-09T20:48:00.923 INFO:teuthology.orchestra.run.smithi067.stderr: 46200K .......... .......... .......... .......... .......... 62% 214M 0s 2019-12-09T20:48:00.924 INFO:teuthology.orchestra.run.smithi067.stderr: 46250K .......... .......... .......... .......... .......... 62% 202M 0s 2019-12-09T20:48:00.924 INFO:teuthology.orchestra.run.smithi067.stderr: 46300K .......... .......... .......... .......... .......... 63% 212M 0s 2019-12-09T20:48:00.924 INFO:teuthology.orchestra.run.smithi067.stderr: 46350K .......... .......... .......... .......... .......... 63% 193M 0s 2019-12-09T20:48:00.924 INFO:teuthology.orchestra.run.smithi067.stderr: 46400K .......... .......... .......... .......... .......... 63% 223M 0s 2019-12-09T20:48:00.924 INFO:teuthology.orchestra.run.smithi067.stderr: 46450K .......... .......... .......... .......... .......... 63% 209M 0s 2019-12-09T20:48:00.924 INFO:teuthology.orchestra.run.smithi067.stderr: 46500K .......... .......... .......... .......... .......... 63% 206M 0s 2019-12-09T20:48:00.924 INFO:teuthology.orchestra.run.smithi067.stderr: 46550K .......... .......... .......... .......... .......... 63% 183M 0s 2019-12-09T20:48:00.925 INFO:teuthology.orchestra.run.smithi067.stderr: 46600K .......... .......... .......... .......... .......... 63% 226M 0s 2019-12-09T20:48:00.925 INFO:teuthology.orchestra.run.smithi067.stderr: 46650K .......... .......... .......... .......... .......... 63% 218M 0s 2019-12-09T20:48:00.925 INFO:teuthology.orchestra.run.smithi067.stderr: 46700K .......... .......... .......... .......... .......... 63% 219M 0s 2019-12-09T20:48:00.925 INFO:teuthology.orchestra.run.smithi067.stderr: 46750K .......... .......... .......... .......... .......... 63% 190M 0s 2019-12-09T20:48:00.925 INFO:teuthology.orchestra.run.smithi067.stderr: 46800K .......... .......... .......... .......... .......... 63% 213M 0s 2019-12-09T20:48:00.925 INFO:teuthology.orchestra.run.smithi067.stderr: 46850K .......... .......... .......... .......... .......... 63% 217M 0s 2019-12-09T20:48:00.925 INFO:teuthology.orchestra.run.smithi067.stderr: 46900K .......... .......... .......... .......... .......... 63% 204M 0s 2019-12-09T20:48:00.925 INFO:teuthology.orchestra.run.smithi067.stderr: 46950K .......... .......... .......... .......... .......... 63% 204M 0s 2019-12-09T20:48:00.926 INFO:teuthology.orchestra.run.smithi067.stderr: 47000K .......... .......... .......... .......... .......... 63% 220M 0s 2019-12-09T20:48:00.926 INFO:teuthology.orchestra.run.smithi067.stderr: 47050K .......... .......... .......... .......... .......... 64% 215M 0s 2019-12-09T20:48:00.926 INFO:teuthology.orchestra.run.smithi067.stderr: 47100K .......... .......... .......... .......... .......... 64% 211M 0s 2019-12-09T20:48:00.926 INFO:teuthology.orchestra.run.smithi067.stderr: 47150K .......... .......... .......... .......... .......... 64% 177M 0s 2019-12-09T20:48:00.926 INFO:teuthology.orchestra.run.smithi067.stderr: 47200K .......... .......... .......... .......... .......... 64% 213M 0s 2019-12-09T20:48:00.926 INFO:teuthology.orchestra.run.smithi067.stderr: 47250K .......... .......... .......... .......... .......... 64% 220M 0s 2019-12-09T20:48:00.926 INFO:teuthology.orchestra.run.smithi067.stderr: 47300K .......... .......... .......... .......... .......... 64% 228M 0s 2019-12-09T20:48:00.927 INFO:teuthology.orchestra.run.smithi067.stderr: 47350K .......... .......... .......... .......... .......... 64% 186M 0s 2019-12-09T20:48:00.927 INFO:teuthology.orchestra.run.smithi067.stderr: 47400K .......... .......... .......... .......... .......... 64% 218M 0s 2019-12-09T20:48:00.927 INFO:teuthology.orchestra.run.smithi067.stderr: 47450K .......... .......... .......... .......... .......... 64% 224M 0s 2019-12-09T20:48:00.927 INFO:teuthology.orchestra.run.smithi067.stderr: 47500K .......... .......... .......... .......... .......... 64% 236M 0s 2019-12-09T20:48:00.927 INFO:teuthology.orchestra.run.smithi067.stderr: 47550K .......... .......... .......... .......... .......... 64% 179M 0s 2019-12-09T20:48:00.927 INFO:teuthology.orchestra.run.smithi067.stderr: 47600K .......... .......... .......... .......... .......... 64% 221M 0s 2019-12-09T20:48:00.927 INFO:teuthology.orchestra.run.smithi067.stderr: 47650K .......... .......... .......... .......... .......... 64% 227M 0s 2019-12-09T20:48:00.928 INFO:teuthology.orchestra.run.smithi067.stderr: 47700K .......... .......... .......... .......... .......... 64% 232M 0s 2019-12-09T20:48:00.928 INFO:teuthology.orchestra.run.smithi067.stderr: 47750K .......... .......... .......... .......... .......... 64% 191M 0s 2019-12-09T20:48:00.928 INFO:teuthology.orchestra.run.smithi067.stderr: 47800K .......... .......... .......... .......... .......... 65% 226M 0s 2019-12-09T20:48:00.928 INFO:teuthology.orchestra.run.smithi067.stderr: 47850K .......... .......... .......... .......... .......... 65% 14.8M 0s 2019-12-09T20:48:00.928 INFO:teuthology.orchestra.run.smithi067.stderr: 47900K .......... .......... .......... .......... .......... 65% 251M 0s 2019-12-09T20:48:00.928 INFO:teuthology.orchestra.run.smithi067.stderr: 47950K .......... .......... .......... .......... .......... 65% 294M 0s 2019-12-09T20:48:00.928 INFO:teuthology.orchestra.run.smithi067.stderr: 48000K .......... .......... .......... .......... .......... 65% 344M 0s 2019-12-09T20:48:00.928 INFO:teuthology.orchestra.run.smithi067.stderr: 48050K .......... .......... .......... .......... .......... 65% 333M 0s 2019-12-09T20:48:00.929 INFO:teuthology.orchestra.run.smithi067.stderr: 48100K .......... .......... .......... .......... .......... 65% 317M 0s 2019-12-09T20:48:00.929 INFO:teuthology.orchestra.run.smithi067.stderr: 48150K .......... .......... .......... .......... .......... 65% 267M 0s 2019-12-09T20:48:00.929 INFO:teuthology.orchestra.run.smithi067.stderr: 48200K .......... .......... .......... .......... .......... 65% 319M 0s 2019-12-09T20:48:00.929 INFO:teuthology.orchestra.run.smithi067.stderr: 48250K .......... .......... .......... .......... .......... 65% 353M 0s 2019-12-09T20:48:00.929 INFO:teuthology.orchestra.run.smithi067.stderr: 48300K .......... .......... .......... .......... .......... 65% 363M 0s 2019-12-09T20:48:00.929 INFO:teuthology.orchestra.run.smithi067.stderr: 48350K .......... .......... .......... .......... .......... 65% 307M 0s 2019-12-09T20:48:00.929 INFO:teuthology.orchestra.run.smithi067.stderr: 48400K .......... .......... .......... .......... .......... 65% 363M 0s 2019-12-09T20:48:00.930 INFO:teuthology.orchestra.run.smithi067.stderr: 48450K .......... .......... .......... .......... .......... 65% 370M 0s 2019-12-09T20:48:00.930 INFO:teuthology.orchestra.run.smithi067.stderr: 48500K .......... .......... .......... .......... .......... 66% 275M 0s 2019-12-09T20:48:00.930 INFO:teuthology.orchestra.run.smithi067.stderr: 48550K .......... .......... .......... .......... .......... 66% 264M 0s 2019-12-09T20:48:00.930 INFO:teuthology.orchestra.run.smithi067.stderr: 48600K .......... .......... .......... .......... .......... 66% 387M 0s 2019-12-09T20:48:00.930 INFO:teuthology.orchestra.run.smithi067.stderr: 48650K .......... .......... .......... .......... .......... 66% 363M 0s 2019-12-09T20:48:00.930 INFO:teuthology.orchestra.run.smithi067.stderr: 48700K .......... .......... .......... .......... .......... 66% 376M 0s 2019-12-09T20:48:00.930 INFO:teuthology.orchestra.run.smithi067.stderr: 48750K .......... .......... .......... .......... .......... 66% 353M 0s 2019-12-09T20:48:00.931 INFO:teuthology.orchestra.run.smithi067.stderr: 48800K .......... .......... .......... .......... .......... 66% 34.3M 0s 2019-12-09T20:48:00.931 INFO:teuthology.orchestra.run.smithi067.stderr: 48850K .......... .......... .......... .......... .......... 66% 349M 0s 2019-12-09T20:48:00.931 INFO:teuthology.orchestra.run.smithi067.stderr: 48900K .......... .......... .......... .......... .......... 66% 352M 0s 2019-12-09T20:48:00.931 INFO:teuthology.orchestra.run.smithi067.stderr: 48950K .......... .......... .......... .......... .......... 66% 284M 0s 2019-12-09T20:48:00.931 INFO:teuthology.orchestra.run.smithi067.stderr: 49000K .......... .......... .......... .......... .......... 66% 361M 0s 2019-12-09T20:48:00.931 INFO:teuthology.orchestra.run.smithi067.stderr: 49050K .......... .......... .......... .......... .......... 66% 359M 0s 2019-12-09T20:48:00.931 INFO:teuthology.orchestra.run.smithi067.stderr: 49100K .......... .......... .......... .......... .......... 66% 358M 0s 2019-12-09T20:48:00.932 INFO:teuthology.orchestra.run.smithi067.stderr: 49150K .......... .......... .......... .......... .......... 66% 336M 0s 2019-12-09T20:48:00.932 INFO:teuthology.orchestra.run.smithi067.stderr: 49200K .......... .......... .......... .......... .......... 66% 357M 0s 2019-12-09T20:48:00.932 INFO:teuthology.orchestra.run.smithi067.stderr: 49250K .......... .......... .......... .......... .......... 67% 351M 0s 2019-12-09T20:48:00.932 INFO:teuthology.orchestra.run.smithi067.stderr: 49300K .......... .......... .......... .......... .......... 67% 326M 0s 2019-12-09T20:48:00.932 INFO:teuthology.orchestra.run.smithi067.stderr: 49350K .......... .......... .......... .......... .......... 67% 270M 0s 2019-12-09T20:48:00.932 INFO:teuthology.orchestra.run.smithi067.stderr: 49400K .......... .......... .......... .......... .......... 67% 324M 0s 2019-12-09T20:48:00.932 INFO:teuthology.orchestra.run.smithi067.stderr: 49450K .......... .......... .......... .......... .......... 67% 347M 0s 2019-12-09T20:48:00.932 INFO:teuthology.orchestra.run.smithi067.stderr: 49500K .......... .......... .......... .......... .......... 67% 341M 0s 2019-12-09T20:48:00.933 INFO:teuthology.orchestra.run.smithi067.stderr: 49550K .......... .......... .......... .......... .......... 67% 307M 0s 2019-12-09T20:48:00.933 INFO:teuthology.orchestra.run.smithi067.stderr: 49600K .......... .......... .......... .......... .......... 67% 316M 0s 2019-12-09T20:48:00.933 INFO:teuthology.orchestra.run.smithi067.stderr: 49650K .......... .......... .......... .......... .......... 67% 353M 0s 2019-12-09T20:48:00.933 INFO:teuthology.orchestra.run.smithi067.stderr: 49700K .......... .......... .......... .......... .......... 67% 354M 0s 2019-12-09T20:48:00.933 INFO:teuthology.orchestra.run.smithi067.stderr: 49750K .......... .......... .......... .......... .......... 67% 335M 0s 2019-12-09T20:48:00.933 INFO:teuthology.orchestra.run.smithi067.stderr: 49800K .......... .......... .......... .......... .......... 67% 389M 0s 2019-12-09T20:48:00.933 INFO:teuthology.orchestra.run.smithi067.stderr: 49850K .......... .......... .......... .......... .......... 67% 367M 0s 2019-12-09T20:48:00.934 INFO:teuthology.orchestra.run.smithi067.stderr: 49900K .......... .......... .......... .......... .......... 67% 378M 0s 2019-12-09T20:48:00.934 INFO:teuthology.orchestra.run.smithi067.stderr: 49950K .......... .......... .......... .......... .......... 67% 304M 0s 2019-12-09T20:48:00.934 INFO:teuthology.orchestra.run.smithi067.stderr: 50000K .......... .......... .......... .......... .......... 68% 360M 0s 2019-12-09T20:48:00.934 INFO:teuthology.orchestra.run.smithi067.stderr: 50050K .......... .......... .......... .......... .......... 68% 386M 0s 2019-12-09T20:48:00.934 INFO:teuthology.orchestra.run.smithi067.stderr: 50100K .......... .......... .......... .......... .......... 68% 376M 0s 2019-12-09T20:48:00.934 INFO:teuthology.orchestra.run.smithi067.stderr: 50150K .......... .......... .......... .......... .......... 68% 327M 0s 2019-12-09T20:48:00.934 INFO:teuthology.orchestra.run.smithi067.stderr: 50200K .......... .......... .......... .......... .......... 68% 386M 0s 2019-12-09T20:48:00.935 INFO:teuthology.orchestra.run.smithi067.stderr: 50250K .......... .......... .......... .......... .......... 68% 390M 0s 2019-12-09T20:48:00.935 INFO:teuthology.orchestra.run.smithi067.stderr: 50300K .......... .......... .......... .......... .......... 68% 373M 0s 2019-12-09T20:48:00.935 INFO:teuthology.orchestra.run.smithi067.stderr: 50350K .......... .......... .......... .......... .......... 68% 273M 0s 2019-12-09T20:48:00.935 INFO:teuthology.orchestra.run.smithi067.stderr: 50400K .......... .......... .......... .......... .......... 68% 378M 0s 2019-12-09T20:48:00.935 INFO:teuthology.orchestra.run.smithi067.stderr: 50450K .......... .......... .......... .......... .......... 68% 375M 0s 2019-12-09T20:48:00.935 INFO:teuthology.orchestra.run.smithi067.stderr: 50500K .......... .......... .......... .......... .......... 68% 334M 0s 2019-12-09T20:48:00.935 INFO:teuthology.orchestra.run.smithi067.stderr: 50550K .......... .......... .......... .......... .......... 68% 312M 0s 2019-12-09T20:48:00.935 INFO:teuthology.orchestra.run.smithi067.stderr: 50600K .......... .......... .......... .......... .......... 68% 343M 0s 2019-12-09T20:48:00.936 INFO:teuthology.orchestra.run.smithi067.stderr: 50650K .......... .......... .......... .......... .......... 68% 395M 0s 2019-12-09T20:48:00.936 INFO:teuthology.orchestra.run.smithi067.stderr: 50700K .......... .......... .......... .......... .......... 68% 365M 0s 2019-12-09T20:48:00.936 INFO:teuthology.orchestra.run.smithi067.stderr: 50750K .......... .......... .......... .......... .......... 69% 318M 0s 2019-12-09T20:48:00.936 INFO:teuthology.orchestra.run.smithi067.stderr: 50800K .......... .......... .......... .......... .......... 69% 410M 0s 2019-12-09T20:48:00.936 INFO:teuthology.orchestra.run.smithi067.stderr: 50850K .......... .......... .......... .......... .......... 69% 402M 0s 2019-12-09T20:48:00.936 INFO:teuthology.orchestra.run.smithi067.stderr: 50900K .......... .......... .......... .......... .......... 69% 134M 0s 2019-12-09T20:48:00.936 INFO:teuthology.orchestra.run.smithi067.stderr: 50950K .......... .......... .......... .......... .......... 69% 300M 0s 2019-12-09T20:48:00.937 INFO:teuthology.orchestra.run.smithi067.stderr: 51000K .......... .......... .......... .......... .......... 69% 356M 0s 2019-12-09T20:48:00.937 INFO:teuthology.orchestra.run.smithi067.stderr: 51050K .......... .......... .......... .......... .......... 69% 373M 0s 2019-12-09T20:48:00.937 INFO:teuthology.orchestra.run.smithi067.stderr: 51100K .......... .......... .......... .......... .......... 69% 361M 0s 2019-12-09T20:48:00.937 INFO:teuthology.orchestra.run.smithi067.stderr: 51150K .......... .......... .......... .......... .......... 69% 308M 0s 2019-12-09T20:48:00.937 INFO:teuthology.orchestra.run.smithi067.stderr: 51200K .......... .......... .......... .......... .......... 69% 361M 0s 2019-12-09T20:48:00.937 INFO:teuthology.orchestra.run.smithi067.stderr: 51250K .......... .......... .......... .......... .......... 69% 361M 0s 2019-12-09T20:48:00.937 INFO:teuthology.orchestra.run.smithi067.stderr: 51300K .......... .......... .......... .......... .......... 69% 370M 0s 2019-12-09T20:48:00.938 INFO:teuthology.orchestra.run.smithi067.stderr: 51350K .......... .......... .......... .......... .......... 69% 318M 0s 2019-12-09T20:48:00.938 INFO:teuthology.orchestra.run.smithi067.stderr: 51400K .......... .......... .......... .......... .......... 69% 399M 0s 2019-12-09T20:48:00.938 INFO:teuthology.orchestra.run.smithi067.stderr: 51450K .......... .......... .......... .......... .......... 70% 376M 0s 2019-12-09T20:48:00.938 INFO:teuthology.orchestra.run.smithi067.stderr: 51500K .......... .......... .......... .......... .......... 70% 359M 0s 2019-12-09T20:48:00.938 INFO:teuthology.orchestra.run.smithi067.stderr: 51550K .......... .......... .......... .......... .......... 70% 291M 0s 2019-12-09T20:48:00.938 INFO:teuthology.orchestra.run.smithi067.stderr: 51600K .......... .......... .......... .......... .......... 70% 350M 0s 2019-12-09T20:48:00.938 INFO:teuthology.orchestra.run.smithi067.stderr: 51650K .......... .......... .......... .......... .......... 70% 310M 0s 2019-12-09T20:48:00.939 INFO:teuthology.orchestra.run.smithi067.stderr: 51700K .......... .......... .......... .......... .......... 70% 321M 0s 2019-12-09T20:48:00.939 INFO:teuthology.orchestra.run.smithi067.stderr: 51750K .......... .......... .......... .......... .......... 70% 296M 0s 2019-12-09T20:48:00.939 INFO:teuthology.orchestra.run.smithi067.stderr: 51800K .......... .......... .......... .......... .......... 70% 331M 0s 2019-12-09T20:48:00.939 INFO:teuthology.orchestra.run.smithi067.stderr: 51850K .......... .......... .......... .......... .......... 70% 361M 0s 2019-12-09T20:48:00.939 INFO:teuthology.orchestra.run.smithi067.stderr: 51900K .......... .......... .......... .......... .......... 70% 379M 0s 2019-12-09T20:48:00.939 INFO:teuthology.orchestra.run.smithi067.stderr: 51950K .......... .......... .......... .......... .......... 70% 315M 0s 2019-12-09T20:48:00.939 INFO:teuthology.orchestra.run.smithi067.stderr: 52000K .......... .......... .......... .......... .......... 70% 360M 0s 2019-12-09T20:48:00.939 INFO:teuthology.orchestra.run.smithi067.stderr: 52050K .......... .......... .......... .......... .......... 70% 367M 0s 2019-12-09T20:48:00.940 INFO:teuthology.orchestra.run.smithi067.stderr: 52100K .......... .......... .......... .......... .......... 70% 395M 0s 2019-12-09T20:48:00.940 INFO:teuthology.orchestra.run.smithi067.stderr: 52150K .......... .......... .......... .......... .......... 70% 286M 0s 2019-12-09T20:48:00.940 INFO:teuthology.orchestra.run.smithi067.stderr: 52200K .......... .......... .......... .......... .......... 71% 242M 0s 2019-12-09T20:48:00.940 INFO:teuthology.orchestra.run.smithi067.stderr: 52250K .......... .......... .......... .......... .......... 71% 319M 0s 2019-12-09T20:48:00.940 INFO:teuthology.orchestra.run.smithi067.stderr: 52300K .......... .......... .......... .......... .......... 71% 304M 0s 2019-12-09T20:48:00.940 INFO:teuthology.orchestra.run.smithi067.stderr: 52350K .......... .......... .......... .......... .......... 71% 283M 0s 2019-12-09T20:48:00.940 INFO:teuthology.orchestra.run.smithi067.stderr: 52400K .......... .......... .......... .......... .......... 71% 392M 0s 2019-12-09T20:48:00.941 INFO:teuthology.orchestra.run.smithi067.stderr: 52450K .......... .......... .......... .......... .......... 71% 15.7M 0s 2019-12-09T20:48:00.941 INFO:teuthology.orchestra.run.smithi067.stderr: 52500K .......... .......... .......... .......... .......... 71% 312M 0s 2019-12-09T20:48:00.941 INFO:teuthology.orchestra.run.smithi067.stderr: 52550K .......... .......... .......... .......... .......... 71% 275M 0s 2019-12-09T20:48:00.941 INFO:teuthology.orchestra.run.smithi067.stderr: 52600K .......... .......... .......... .......... .......... 71% 306M 0s 2019-12-09T20:48:00.941 INFO:teuthology.orchestra.run.smithi067.stderr: 52650K .......... .......... .......... .......... .......... 71% 370M 0s 2019-12-09T20:48:00.941 INFO:teuthology.orchestra.run.smithi067.stderr: 52700K .......... .......... .......... .......... .......... 71% 344M 0s 2019-12-09T20:48:00.941 INFO:teuthology.orchestra.run.smithi067.stderr: 52750K .......... .......... .......... .......... .......... 71% 311M 0s 2019-12-09T20:48:00.942 INFO:teuthology.orchestra.run.smithi067.stderr: 52800K .......... .......... .......... .......... .......... 71% 356M 0s 2019-12-09T20:48:00.942 INFO:teuthology.orchestra.run.smithi067.stderr: 52850K .......... .......... .......... .......... .......... 71% 307M 0s 2019-12-09T20:48:00.942 INFO:teuthology.orchestra.run.smithi067.stderr: 52900K .......... .......... .......... .......... .......... 71% 341M 0s 2019-12-09T20:48:00.942 INFO:teuthology.orchestra.run.smithi067.stderr: 52950K .......... .......... .......... .......... .......... 72% 292M 0s 2019-12-09T20:48:00.942 INFO:teuthology.orchestra.run.smithi067.stderr: 53000K .......... .......... .......... .......... .......... 72% 346M 0s 2019-12-09T20:48:00.942 INFO:teuthology.orchestra.run.smithi067.stderr: 53050K .......... .......... .......... .......... .......... 72% 331M 0s 2019-12-09T20:48:00.942 INFO:teuthology.orchestra.run.smithi067.stderr: 53100K .......... .......... .......... .......... .......... 72% 340M 0s 2019-12-09T20:48:00.942 INFO:teuthology.orchestra.run.smithi067.stderr: 53150K .......... .......... .......... .......... .......... 72% 325M 0s 2019-12-09T20:48:00.943 INFO:teuthology.orchestra.run.smithi067.stderr: 53200K .......... .......... .......... .......... .......... 72% 354M 0s 2019-12-09T20:48:00.943 INFO:teuthology.orchestra.run.smithi067.stderr: 53250K .......... .......... .......... .......... .......... 72% 377M 0s 2019-12-09T20:48:00.943 INFO:teuthology.orchestra.run.smithi067.stderr: 53300K .......... .......... .......... .......... .......... 72% 374M 0s 2019-12-09T20:48:00.943 INFO:teuthology.orchestra.run.smithi067.stderr: 53350K .......... .......... .......... .......... .......... 72% 350M 0s 2019-12-09T20:48:00.943 INFO:teuthology.orchestra.run.smithi067.stderr: 53400K .......... .......... .......... .......... .......... 72% 87.9M 0s 2019-12-09T20:48:00.943 INFO:teuthology.orchestra.run.smithi067.stderr: 53450K .......... .......... .......... .......... .......... 72% 280M 0s 2019-12-09T20:48:00.943 INFO:teuthology.orchestra.run.smithi067.stderr: 53500K .......... .......... .......... .......... .......... 72% 281M 0s 2019-12-09T20:48:00.944 INFO:teuthology.orchestra.run.smithi067.stderr: 53550K .......... .......... .......... .......... .......... 72% 255M 0s 2019-12-09T20:48:00.944 INFO:teuthology.orchestra.run.smithi067.stderr: 53600K .......... .......... .......... .......... .......... 72% 300M 0s 2019-12-09T20:48:00.944 INFO:teuthology.orchestra.run.smithi067.stderr: 53650K .......... .......... .......... .......... .......... 73% 329M 0s 2019-12-09T20:48:00.944 INFO:teuthology.orchestra.run.smithi067.stderr: 53700K .......... .......... .......... .......... .......... 73% 350M 0s 2019-12-09T20:48:00.944 INFO:teuthology.orchestra.run.smithi067.stderr: 53750K .......... .......... .......... .......... .......... 73% 341M 0s 2019-12-09T20:48:00.944 INFO:teuthology.orchestra.run.smithi067.stderr: 53800K .......... .......... .......... .......... .......... 73% 385M 0s 2019-12-09T20:48:00.944 INFO:teuthology.orchestra.run.smithi067.stderr: 53850K .......... .......... .......... .......... .......... 73% 398M 0s 2019-12-09T20:48:00.945 INFO:teuthology.orchestra.run.smithi067.stderr: 53900K .......... .......... .......... .......... .......... 73% 45.3M 0s 2019-12-09T20:48:00.945 INFO:teuthology.orchestra.run.smithi067.stderr: 53950K .......... .......... .......... .......... .......... 73% 172M 0s 2019-12-09T20:48:00.945 INFO:teuthology.orchestra.run.smithi067.stderr: 54000K .......... .......... .......... .......... .......... 73% 282M 0s 2019-12-09T20:48:00.945 INFO:teuthology.orchestra.run.smithi067.stderr: 54050K .......... .......... .......... .......... .......... 73% 310M 0s 2019-12-09T20:48:00.945 INFO:teuthology.orchestra.run.smithi067.stderr: 54100K .......... .......... .......... .......... .......... 73% 361M 0s 2019-12-09T20:48:00.945 INFO:teuthology.orchestra.run.smithi067.stderr: 54150K .......... .......... .......... .......... .......... 73% 347M 0s 2019-12-09T20:48:00.945 INFO:teuthology.orchestra.run.smithi067.stderr: 54200K .......... .......... .......... .......... .......... 73% 400M 0s 2019-12-09T20:48:00.946 INFO:teuthology.orchestra.run.smithi067.stderr: 54250K .......... .......... .......... .......... .......... 73% 12.3M 0s 2019-12-09T20:48:00.946 INFO:teuthology.orchestra.run.smithi067.stderr: 54300K .......... .......... .......... .......... .......... 73% 346M 0s 2019-12-09T20:48:00.946 INFO:teuthology.orchestra.run.smithi067.stderr: 54350K .......... .......... .......... .......... .......... 73% 317M 0s 2019-12-09T20:48:00.946 INFO:teuthology.orchestra.run.smithi067.stderr: 54400K .......... .......... .......... .......... .......... 74% 368M 0s 2019-12-09T20:48:00.946 INFO:teuthology.orchestra.run.smithi067.stderr: 54450K .......... .......... .......... .......... .......... 74% 354M 0s 2019-12-09T20:48:00.946 INFO:teuthology.orchestra.run.smithi067.stderr: 54500K .......... .......... .......... .......... .......... 74% 355M 0s 2019-12-09T20:48:00.946 INFO:teuthology.orchestra.run.smithi067.stderr: 54550K .......... .......... .......... .......... .......... 74% 292M 0s 2019-12-09T20:48:00.946 INFO:teuthology.orchestra.run.smithi067.stderr: 54600K .......... .......... .......... .......... .......... 74% 343M 0s 2019-12-09T20:48:00.947 INFO:teuthology.orchestra.run.smithi067.stderr: 54650K .......... .......... .......... .......... .......... 74% 340M 0s 2019-12-09T20:48:00.947 INFO:teuthology.orchestra.run.smithi067.stderr: 54700K .......... .......... .......... .......... .......... 74% 367M 0s 2019-12-09T20:48:00.947 INFO:teuthology.orchestra.run.smithi067.stderr: 54750K .......... .......... .......... .......... .......... 74% 321M 0s 2019-12-09T20:48:00.947 INFO:teuthology.orchestra.run.smithi067.stderr: 54800K .......... .......... .......... .......... .......... 74% 356M 0s 2019-12-09T20:48:00.947 INFO:teuthology.orchestra.run.smithi067.stderr: 54850K .......... .......... .......... .......... .......... 74% 295M 0s 2019-12-09T20:48:00.947 INFO:teuthology.orchestra.run.smithi067.stderr: 54900K .......... .......... .......... .......... .......... 74% 303M 0s 2019-12-09T20:48:00.947 INFO:teuthology.orchestra.run.smithi067.stderr: 54950K .......... .......... .......... .......... .......... 74% 262M 0s 2019-12-09T20:48:00.948 INFO:teuthology.orchestra.run.smithi067.stderr: 55000K .......... .......... .......... .......... .......... 74% 326M 0s 2019-12-09T20:48:00.948 INFO:teuthology.orchestra.run.smithi067.stderr: 55050K .......... .......... .......... .......... .......... 74% 371M 0s 2019-12-09T20:48:00.948 INFO:teuthology.orchestra.run.smithi067.stderr: 55100K .......... .......... .......... .......... .......... 74% 387M 0s 2019-12-09T20:48:00.948 INFO:teuthology.orchestra.run.smithi067.stderr: 55150K .......... .......... .......... .......... .......... 75% 336M 0s 2019-12-09T20:48:00.948 INFO:teuthology.orchestra.run.smithi067.stderr: 55200K .......... .......... .......... .......... .......... 75% 392M 0s 2019-12-09T20:48:00.948 INFO:teuthology.orchestra.run.smithi067.stderr: 55250K .......... .......... .......... .......... .......... 75% 390M 0s 2019-12-09T20:48:00.948 INFO:teuthology.orchestra.run.smithi067.stderr: 55300K .......... .......... .......... .......... .......... 75% 361M 0s 2019-12-09T20:48:00.949 INFO:teuthology.orchestra.run.smithi067.stderr: 55350K .......... .......... .......... .......... .......... 75% 308M 0s 2019-12-09T20:48:00.949 INFO:teuthology.orchestra.run.smithi067.stderr: 55400K .......... .......... .......... .......... .......... 75% 388M 0s 2019-12-09T20:48:00.949 INFO:teuthology.orchestra.run.smithi067.stderr: 55450K .......... .......... .......... .......... .......... 75% 389M 0s 2019-12-09T20:48:00.949 INFO:teuthology.orchestra.run.smithi067.stderr: 55500K .......... .......... .......... .......... .......... 75% 404M 0s 2019-12-09T20:48:00.949 INFO:teuthology.orchestra.run.smithi067.stderr: 55550K .......... .......... .......... .......... .......... 75% 352M 0s 2019-12-09T20:48:00.949 INFO:teuthology.orchestra.run.smithi067.stderr: 55600K .......... .......... .......... .......... .......... 75% 400M 0s 2019-12-09T20:48:00.949 INFO:teuthology.orchestra.run.smithi067.stderr: 55650K .......... .......... .......... .......... .......... 75% 399M 0s 2019-12-09T20:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr: 55700K .......... .......... .......... .......... .......... 75% 381M 0s 2019-12-09T20:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr: 55750K .......... .......... .......... .......... .......... 75% 199M 0s 2019-12-09T20:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr: 55800K .......... .......... .......... .......... .......... 75% 347M 0s 2019-12-09T20:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr: 55850K .......... .......... .......... .......... .......... 75% 362M 0s 2019-12-09T20:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr: 55900K .......... .......... .......... .......... .......... 76% 375M 0s 2019-12-09T20:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr: 55950K .......... .......... .......... .......... .......... 76% 331M 0s 2019-12-09T20:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr: 56000K .......... .......... .......... .......... .......... 76% 368M 0s 2019-12-09T20:48:00.950 INFO:teuthology.orchestra.run.smithi067.stderr: 56050K .......... .......... .......... .......... .......... 76% 356M 0s 2019-12-09T20:48:00.951 INFO:teuthology.orchestra.run.smithi067.stderr: 56100K .......... .......... .......... .......... .......... 76% 310M 0s 2019-12-09T20:48:00.951 INFO:teuthology.orchestra.run.smithi067.stderr: 56150K .......... .......... .......... .......... .......... 76% 278M 0s 2019-12-09T20:48:00.951 INFO:teuthology.orchestra.run.smithi067.stderr: 56200K .......... .......... .......... .......... .......... 76% 343M 0s 2019-12-09T20:48:00.951 INFO:teuthology.orchestra.run.smithi067.stderr: 56250K .......... .......... .......... .......... .......... 76% 342M 0s 2019-12-09T20:48:00.951 INFO:teuthology.orchestra.run.smithi067.stderr: 56300K .......... .......... .......... .......... .......... 76% 346M 0s 2019-12-09T20:48:00.951 INFO:teuthology.orchestra.run.smithi067.stderr: 56350K .......... .......... .......... .......... .......... 76% 320M 0s 2019-12-09T20:48:00.951 INFO:teuthology.orchestra.run.smithi067.stderr: 56400K .......... .......... .......... .......... .......... 76% 389M 0s 2019-12-09T20:48:00.952 INFO:teuthology.orchestra.run.smithi067.stderr: 56450K .......... .......... .......... .......... .......... 76% 415M 0s 2019-12-09T20:48:00.952 INFO:teuthology.orchestra.run.smithi067.stderr: 56500K .......... .......... .......... .......... .......... 76% 408M 0s 2019-12-09T20:48:00.952 INFO:teuthology.orchestra.run.smithi067.stderr: 56550K .......... .......... .......... .......... .......... 76% 338M 0s 2019-12-09T20:48:00.952 INFO:teuthology.orchestra.run.smithi067.stderr: 56600K .......... .......... .......... .......... .......... 77% 406M 0s 2019-12-09T20:48:00.952 INFO:teuthology.orchestra.run.smithi067.stderr: 56650K .......... .......... .......... .......... .......... 77% 412M 0s 2019-12-09T20:48:00.952 INFO:teuthology.orchestra.run.smithi067.stderr: 56700K .......... .......... .......... .......... .......... 77% 129M 0s 2019-12-09T20:48:00.952 INFO:teuthology.orchestra.run.smithi067.stderr: 56750K .......... .......... .......... .......... .......... 77% 275M 0s 2019-12-09T20:48:00.953 INFO:teuthology.orchestra.run.smithi067.stderr: 56800K .......... .......... .......... .......... .......... 77% 292M 0s 2019-12-09T20:48:00.953 INFO:teuthology.orchestra.run.smithi067.stderr: 56850K .......... .......... .......... .......... .......... 77% 313M 0s 2019-12-09T20:48:00.953 INFO:teuthology.orchestra.run.smithi067.stderr: 56900K .......... .......... .......... .......... .......... 77% 393M 0s 2019-12-09T20:48:00.953 INFO:teuthology.orchestra.run.smithi067.stderr: 56950K .......... .......... .......... .......... .......... 77% 165M 0s 2019-12-09T20:48:00.953 INFO:teuthology.orchestra.run.smithi067.stderr: 57000K .......... .......... .......... .......... .......... 77% 206M 0s 2019-12-09T20:48:00.953 INFO:teuthology.orchestra.run.smithi067.stderr: 57050K .......... .......... .......... .......... .......... 77% 279M 0s 2019-12-09T20:48:00.953 INFO:teuthology.orchestra.run.smithi067.stderr: 57100K .......... .......... .......... .......... .......... 77% 391M 0s 2019-12-09T20:48:00.953 INFO:teuthology.orchestra.run.smithi067.stderr: 57150K .......... .......... .......... .......... .......... 77% 355M 0s 2019-12-09T20:48:00.954 INFO:teuthology.orchestra.run.smithi067.stderr: 57200K .......... .......... .......... .......... .......... 77% 405M 0s 2019-12-09T20:48:00.954 INFO:teuthology.orchestra.run.smithi067.stderr: 57250K .......... .......... .......... .......... .......... 77% 143M 0s 2019-12-09T20:48:00.954 INFO:teuthology.orchestra.run.smithi067.stderr: 57300K .......... .......... .......... .......... .......... 77% 297M 0s 2019-12-09T20:48:00.954 INFO:teuthology.orchestra.run.smithi067.stderr: 57350K .......... .......... .......... .......... .......... 78% 268M 0s 2019-12-09T20:48:00.954 INFO:teuthology.orchestra.run.smithi067.stderr: 57400K .......... .......... .......... .......... .......... 78% 321M 0s 2019-12-09T20:48:00.954 INFO:teuthology.orchestra.run.smithi067.stderr: 57450K .......... .......... .......... .......... .......... 78% 417M 0s 2019-12-09T20:48:00.954 INFO:teuthology.orchestra.run.smithi067.stderr: 57500K .......... .......... .......... .......... .......... 78% 425M 0s 2019-12-09T20:48:00.955 INFO:teuthology.orchestra.run.smithi067.stderr: 57550K .......... .......... .......... .......... .......... 78% 53.6M 0s 2019-12-09T20:48:00.955 INFO:teuthology.orchestra.run.smithi067.stderr: 57600K .......... .......... .......... .......... .......... 78% 327M 0s 2019-12-09T20:48:00.955 INFO:teuthology.orchestra.run.smithi067.stderr: 57650K .......... .......... .......... .......... .......... 78% 318M 0s 2019-12-09T20:48:00.955 INFO:teuthology.orchestra.run.smithi067.stderr: 57700K .......... .......... .......... .......... .......... 78% 343M 0s 2019-12-09T20:48:00.955 INFO:teuthology.orchestra.run.smithi067.stderr: 57750K .......... .......... .......... .......... .......... 78% 289M 0s 2019-12-09T20:48:00.955 INFO:teuthology.orchestra.run.smithi067.stderr: 57800K .......... .......... .......... .......... .......... 78% 340M 0s 2019-12-09T20:48:00.955 INFO:teuthology.orchestra.run.smithi067.stderr: 57850K .......... .......... .......... .......... .......... 78% 364M 0s 2019-12-09T20:48:00.956 INFO:teuthology.orchestra.run.smithi067.stderr: 57900K .......... .......... .......... .......... .......... 78% 337M 0s 2019-12-09T20:48:00.956 INFO:teuthology.orchestra.run.smithi067.stderr: 57950K .......... .......... .......... .......... .......... 78% 305M 0s 2019-12-09T20:48:00.956 INFO:teuthology.orchestra.run.smithi067.stderr: 58000K .......... .......... .......... .......... .......... 78% 349M 0s 2019-12-09T20:48:00.956 INFO:teuthology.orchestra.run.smithi067.stderr: 58050K .......... .......... .......... .......... .......... 78% 335M 0s 2019-12-09T20:48:00.956 INFO:teuthology.orchestra.run.smithi067.stderr: 58100K .......... .......... .......... .......... .......... 79% 376M 0s 2019-12-09T20:48:00.956 INFO:teuthology.orchestra.run.smithi067.stderr: 58150K .......... .......... .......... .......... .......... 79% 309M 0s 2019-12-09T20:48:00.956 INFO:teuthology.orchestra.run.smithi067.stderr: 58200K .......... .......... .......... .......... .......... 79% 390M 0s 2019-12-09T20:48:00.957 INFO:teuthology.orchestra.run.smithi067.stderr: 58250K .......... .......... .......... .......... .......... 79% 361M 0s 2019-12-09T20:48:00.957 INFO:teuthology.orchestra.run.smithi067.stderr: 58300K .......... .......... .......... .......... .......... 79% 365M 0s 2019-12-09T20:48:00.957 INFO:teuthology.orchestra.run.smithi067.stderr: 58350K .......... .......... .......... .......... .......... 79% 326M 0s 2019-12-09T20:48:00.957 INFO:teuthology.orchestra.run.smithi067.stderr: 58400K .......... .......... .......... .......... .......... 79% 380M 0s 2019-12-09T20:48:00.957 INFO:teuthology.orchestra.run.smithi067.stderr: 58450K .......... .......... .......... .......... .......... 79% 402M 0s 2019-12-09T20:48:00.957 INFO:teuthology.orchestra.run.smithi067.stderr: 58500K .......... .......... .......... .......... .......... 79% 406M 0s 2019-12-09T20:48:00.957 INFO:teuthology.orchestra.run.smithi067.stderr: 58550K .......... .......... .......... .......... .......... 79% 346M 0s 2019-12-09T20:48:00.958 INFO:teuthology.orchestra.run.smithi067.stderr: 58600K .......... .......... .......... .......... .......... 79% 411M 0s 2019-12-09T20:48:00.958 INFO:teuthology.orchestra.run.smithi067.stderr: 58650K .......... .......... .......... .......... .......... 79% 402M 0s 2019-12-09T20:48:00.958 INFO:teuthology.orchestra.run.smithi067.stderr: 58700K .......... .......... .......... .......... .......... 79% 54.8M 0s 2019-12-09T20:48:00.958 INFO:teuthology.orchestra.run.smithi067.stderr: 58750K .......... .......... .......... .......... .......... 79% 208M 0s 2019-12-09T20:48:00.958 INFO:teuthology.orchestra.run.smithi067.stderr: 58800K .......... .......... .......... .......... .......... 80% 280M 0s 2019-12-09T20:48:00.958 INFO:teuthology.orchestra.run.smithi067.stderr: 58850K .......... .......... .......... .......... .......... 80% 370M 0s 2019-12-09T20:48:00.958 INFO:teuthology.orchestra.run.smithi067.stderr: 58900K .......... .......... .......... .......... .......... 80% 364M 0s 2019-12-09T20:48:00.959 INFO:teuthology.orchestra.run.smithi067.stderr: 58950K .......... .......... .......... .......... .......... 80% 318M 0s 2019-12-09T20:48:00.959 INFO:teuthology.orchestra.run.smithi067.stderr: 59000K .......... .......... .......... .......... .......... 80% 363M 0s 2019-12-09T20:48:00.959 INFO:teuthology.orchestra.run.smithi067.stderr: 59050K .......... .......... .......... .......... .......... 80% 372M 0s 2019-12-09T20:48:00.959 INFO:teuthology.orchestra.run.smithi067.stderr: 59100K .......... .......... .......... .......... .......... 80% 355M 0s 2019-12-09T20:48:00.959 INFO:teuthology.orchestra.run.smithi067.stderr: 59150K .......... .......... .......... .......... .......... 80% 342M 0s 2019-12-09T20:48:00.959 INFO:teuthology.orchestra.run.smithi067.stderr: 59200K .......... .......... .......... .......... .......... 80% 390M 0s 2019-12-09T20:48:00.959 INFO:teuthology.orchestra.run.smithi067.stderr: 59250K .......... .......... .......... .......... .......... 80% 12.1M 0s 2019-12-09T20:48:00.959 INFO:teuthology.orchestra.run.smithi067.stderr: 59300K .......... .......... .......... .......... .......... 80% 236M 0s 2019-12-09T20:48:00.960 INFO:teuthology.orchestra.run.smithi067.stderr: 59350K .......... .......... .......... .......... .......... 80% 186M 0s 2019-12-09T20:48:00.960 INFO:teuthology.orchestra.run.smithi067.stderr: 59400K .......... .......... .......... .......... .......... 80% 270M 0s 2019-12-09T20:48:00.960 INFO:teuthology.orchestra.run.smithi067.stderr: 59450K .......... .......... .......... .......... .......... 80% 274M 0s 2019-12-09T20:48:00.960 INFO:teuthology.orchestra.run.smithi067.stderr: 59500K .......... .......... .......... .......... .......... 80% 298M 0s 2019-12-09T20:48:00.960 INFO:teuthology.orchestra.run.smithi067.stderr: 59550K .......... .......... .......... .......... .......... 81% 307M 0s 2019-12-09T20:48:00.960 INFO:teuthology.orchestra.run.smithi067.stderr: 59600K .......... .......... .......... .......... .......... 81% 272M 0s 2019-12-09T20:48:00.960 INFO:teuthology.orchestra.run.smithi067.stderr: 59650K .......... .......... .......... .......... .......... 81% 309M 0s 2019-12-09T20:48:00.961 INFO:teuthology.orchestra.run.smithi067.stderr: 59700K .......... .......... .......... .......... .......... 81% 314M 0s 2019-12-09T20:48:00.961 INFO:teuthology.orchestra.run.smithi067.stderr: 59750K .......... .......... .......... .......... .......... 81% 255M 0s 2019-12-09T20:48:00.961 INFO:teuthology.orchestra.run.smithi067.stderr: 59800K .......... .......... .......... .......... .......... 81% 357M 0s 2019-12-09T20:48:00.961 INFO:teuthology.orchestra.run.smithi067.stderr: 59850K .......... .......... .......... .......... .......... 81% 374M 0s 2019-12-09T20:48:00.961 INFO:teuthology.orchestra.run.smithi067.stderr: 59900K .......... .......... .......... .......... .......... 81% 353M 0s 2019-12-09T20:48:00.961 INFO:teuthology.orchestra.run.smithi067.stderr: 59950K .......... .......... .......... .......... .......... 81% 300M 0s 2019-12-09T20:48:00.961 INFO:teuthology.orchestra.run.smithi067.stderr: 60000K .......... .......... .......... .......... .......... 81% 317M 0s 2019-12-09T20:48:00.962 INFO:teuthology.orchestra.run.smithi067.stderr: 60050K .......... .......... .......... .......... .......... 81% 327M 0s 2019-12-09T20:48:00.962 INFO:teuthology.orchestra.run.smithi067.stderr: 60100K .......... .......... .......... .......... .......... 81% 333M 0s 2019-12-09T20:48:00.962 INFO:teuthology.orchestra.run.smithi067.stderr: 60150K .......... .......... .......... .......... .......... 81% 270M 0s 2019-12-09T20:48:00.962 INFO:teuthology.orchestra.run.smithi067.stderr: 60200K .......... .......... .......... .......... .......... 81% 313M 0s 2019-12-09T20:48:00.962 INFO:teuthology.orchestra.run.smithi067.stderr: 60250K .......... .......... .......... .......... .......... 81% 352M 0s 2019-12-09T20:48:00.962 INFO:teuthology.orchestra.run.smithi067.stderr: 60300K .......... .......... .......... .......... .......... 82% 301M 0s 2019-12-09T20:48:00.962 INFO:teuthology.orchestra.run.smithi067.stderr: 60350K .......... .......... .......... .......... .......... 82% 264M 0s 2019-12-09T20:48:00.963 INFO:teuthology.orchestra.run.smithi067.stderr: 60400K .......... .......... .......... .......... .......... 82% 318M 0s 2019-12-09T20:48:00.963 INFO:teuthology.orchestra.run.smithi067.stderr: 60450K .......... .......... .......... .......... .......... 82% 330M 0s 2019-12-09T20:48:00.963 INFO:teuthology.orchestra.run.smithi067.stderr: 60500K .......... .......... .......... .......... .......... 82% 349M 0s 2019-12-09T20:48:00.963 INFO:teuthology.orchestra.run.smithi067.stderr: 60550K .......... .......... .......... .......... .......... 82% 303M 0s 2019-12-09T20:48:00.963 INFO:teuthology.orchestra.run.smithi067.stderr: 60600K .......... .......... .......... .......... .......... 82% 360M 0s 2019-12-09T20:48:00.963 INFO:teuthology.orchestra.run.smithi067.stderr: 60650K .......... .......... .......... .......... .......... 82% 301M 0s 2019-12-09T20:48:00.963 INFO:teuthology.orchestra.run.smithi067.stderr: 60700K .......... .......... .......... .......... .......... 82% 328M 0s 2019-12-09T20:48:00.963 INFO:teuthology.orchestra.run.smithi067.stderr: 60750K .......... .......... .......... .......... .......... 82% 331M 0s 2019-12-09T20:48:00.964 INFO:teuthology.orchestra.run.smithi067.stderr: 60800K .......... .......... .......... .......... .......... 82% 364M 0s 2019-12-09T20:48:00.964 INFO:teuthology.orchestra.run.smithi067.stderr: 60850K .......... .......... .......... .......... .......... 82% 358M 0s 2019-12-09T20:48:00.964 INFO:teuthology.orchestra.run.smithi067.stderr: 60900K .......... .......... .......... .......... .......... 82% 363M 0s 2019-12-09T20:48:00.964 INFO:teuthology.orchestra.run.smithi067.stderr: 60950K .......... .......... .......... .......... .......... 82% 310M 0s 2019-12-09T20:48:00.964 INFO:teuthology.orchestra.run.smithi067.stderr: 61000K .......... .......... .......... .......... .......... 82% 373M 0s 2019-12-09T20:48:00.964 INFO:teuthology.orchestra.run.smithi067.stderr: 61050K .......... .......... .......... .......... .......... 83% 355M 0s 2019-12-09T20:48:00.964 INFO:teuthology.orchestra.run.smithi067.stderr: 61100K .......... .......... .......... .......... .......... 83% 375M 0s 2019-12-09T20:48:00.965 INFO:teuthology.orchestra.run.smithi067.stderr: 61150K .......... .......... .......... .......... .......... 83% 346M 0s 2019-12-09T20:48:00.965 INFO:teuthology.orchestra.run.smithi067.stderr: 61200K .......... .......... .......... .......... .......... 83% 379M 0s 2019-12-09T20:48:00.965 INFO:teuthology.orchestra.run.smithi067.stderr: 61250K .......... .......... .......... .......... .......... 83% 382M 0s 2019-12-09T20:48:00.965 INFO:teuthology.orchestra.run.smithi067.stderr: 61300K .......... .......... .......... .......... .......... 83% 384M 0s 2019-12-09T20:48:00.965 INFO:teuthology.orchestra.run.smithi067.stderr: 61350K .......... .......... .......... .......... .......... 83% 308M 0s 2019-12-09T20:48:00.965 INFO:teuthology.orchestra.run.smithi067.stderr: 61400K .......... .......... .......... .......... .......... 83% 352M 0s 2019-12-09T20:48:00.965 INFO:teuthology.orchestra.run.smithi067.stderr: 61450K .......... .......... .......... .......... .......... 83% 386M 0s 2019-12-09T20:48:00.966 INFO:teuthology.orchestra.run.smithi067.stderr: 61500K .......... .......... .......... .......... .......... 83% 388M 0s 2019-12-09T20:48:00.966 INFO:teuthology.orchestra.run.smithi067.stderr: 61550K .......... .......... .......... .......... .......... 83% 345M 0s 2019-12-09T20:48:00.966 INFO:teuthology.orchestra.run.smithi067.stderr: 61600K .......... .......... .......... .......... .......... 83% 401M 0s 2019-12-09T20:48:00.966 INFO:teuthology.orchestra.run.smithi067.stderr: 61650K .......... .......... .......... .......... .......... 83% 403M 0s 2019-12-09T20:48:00.966 INFO:teuthology.orchestra.run.smithi067.stderr: 61700K .......... .......... .......... .......... .......... 83% 372M 0s 2019-12-09T20:48:00.966 INFO:teuthology.orchestra.run.smithi067.stderr: 61750K .......... .......... .......... .......... .......... 84% 333M 0s 2019-12-09T20:48:00.966 INFO:teuthology.orchestra.run.smithi067.stderr: 61800K .......... .......... .......... .......... .......... 84% 363M 0s 2019-12-09T20:48:00.966 INFO:teuthology.orchestra.run.smithi067.stderr: 61850K .......... .......... .......... .......... .......... 84% 385M 0s 2019-12-09T20:48:00.967 INFO:teuthology.orchestra.run.smithi067.stderr: 61900K .......... .......... .......... .......... .......... 84% 379M 0s 2019-12-09T20:48:00.967 INFO:teuthology.orchestra.run.smithi067.stderr: 61950K .......... .......... .......... .......... .......... 84% 361M 0s 2019-12-09T20:48:00.967 INFO:teuthology.orchestra.run.smithi067.stderr: 62000K .......... .......... .......... .......... .......... 84% 392M 0s 2019-12-09T20:48:00.967 INFO:teuthology.orchestra.run.smithi067.stderr: 62050K .......... .......... .......... .......... .......... 84% 409M 0s 2019-12-09T20:48:00.967 INFO:teuthology.orchestra.run.smithi067.stderr: 62100K .......... .......... .......... .......... .......... 84% 408M 0s 2019-12-09T20:48:00.967 INFO:teuthology.orchestra.run.smithi067.stderr: 62150K .......... .......... .......... .......... .......... 84% 331M 0s 2019-12-09T20:48:00.967 INFO:teuthology.orchestra.run.smithi067.stderr: 62200K .......... .......... .......... .......... .......... 84% 154M 0s 2019-12-09T20:48:00.968 INFO:teuthology.orchestra.run.smithi067.stderr: 62250K .......... .......... .......... .......... .......... 84% 268M 0s 2019-12-09T20:48:00.968 INFO:teuthology.orchestra.run.smithi067.stderr: 62300K .......... .......... .......... .......... .......... 84% 334M 0s 2019-12-09T20:48:00.968 INFO:teuthology.orchestra.run.smithi067.stderr: 62350K .......... .......... .......... .......... .......... 84% 319M 0s 2019-12-09T20:48:00.968 INFO:teuthology.orchestra.run.smithi067.stderr: 62400K .......... .......... .......... .......... .......... 84% 390M 0s 2019-12-09T20:48:00.968 INFO:teuthology.orchestra.run.smithi067.stderr: 62450K .......... .......... .......... .......... .......... 84% 338M 0s 2019-12-09T20:48:00.968 INFO:teuthology.orchestra.run.smithi067.stderr: 62500K .......... .......... .......... .......... .......... 85% 324M 0s 2019-12-09T20:48:00.968 INFO:teuthology.orchestra.run.smithi067.stderr: 62550K .......... .......... .......... .......... .......... 85% 305M 0s 2019-12-09T20:48:00.969 INFO:teuthology.orchestra.run.smithi067.stderr: 62600K .......... .......... .......... .......... .......... 85% 353M 0s 2019-12-09T20:48:00.969 INFO:teuthology.orchestra.run.smithi067.stderr: 62650K .......... .......... .......... .......... .......... 85% 356M 0s 2019-12-09T20:48:00.969 INFO:teuthology.orchestra.run.smithi067.stderr: 62700K .......... .......... .......... .......... .......... 85% 351M 0s 2019-12-09T20:48:00.969 INFO:teuthology.orchestra.run.smithi067.stderr: 62750K .......... .......... .......... .......... .......... 85% 299M 0s 2019-12-09T20:48:00.969 INFO:teuthology.orchestra.run.smithi067.stderr: 62800K .......... .......... .......... .......... .......... 85% 356M 0s 2019-12-09T20:48:00.969 INFO:teuthology.orchestra.run.smithi067.stderr: 62850K .......... .......... .......... .......... .......... 85% 362M 0s 2019-12-09T20:48:00.969 INFO:teuthology.orchestra.run.smithi067.stderr: 62900K .......... .......... .......... .......... .......... 85% 378M 0s 2019-12-09T20:48:00.969 INFO:teuthology.orchestra.run.smithi067.stderr: 62950K .......... .......... .......... .......... .......... 85% 322M 0s 2019-12-09T20:48:00.970 INFO:teuthology.orchestra.run.smithi067.stderr: 63000K .......... .......... .......... .......... .......... 85% 391M 0s 2019-12-09T20:48:00.970 INFO:teuthology.orchestra.run.smithi067.stderr: 63050K .......... .......... .......... .......... .......... 85% 395M 0s 2019-12-09T20:48:00.970 INFO:teuthology.orchestra.run.smithi067.stderr: 63100K .......... .......... .......... .......... .......... 85% 375M 0s 2019-12-09T20:48:00.970 INFO:teuthology.orchestra.run.smithi067.stderr: 63150K .......... .......... .......... .......... .......... 85% 333M 0s 2019-12-09T20:48:00.970 INFO:teuthology.orchestra.run.smithi067.stderr: 63200K .......... .......... .......... .......... .......... 85% 346M 0s 2019-12-09T20:48:00.970 INFO:teuthology.orchestra.run.smithi067.stderr: 63250K .......... .......... .......... .......... .......... 86% 358M 0s 2019-12-09T20:48:00.970 INFO:teuthology.orchestra.run.smithi067.stderr: 63300K .......... .......... .......... .......... .......... 86% 388M 0s 2019-12-09T20:48:00.971 INFO:teuthology.orchestra.run.smithi067.stderr: 63350K .......... .......... .......... .......... .......... 86% 302M 0s 2019-12-09T20:48:00.971 INFO:teuthology.orchestra.run.smithi067.stderr: 63400K .......... .......... .......... .......... .......... 86% 342M 0s 2019-12-09T20:48:00.971 INFO:teuthology.orchestra.run.smithi067.stderr: 63450K .......... .......... .......... .......... .......... 86% 356M 0s 2019-12-09T20:48:00.971 INFO:teuthology.orchestra.run.smithi067.stderr: 63500K .......... .......... .......... .......... .......... 86% 370M 0s 2019-12-09T20:48:00.971 INFO:teuthology.orchestra.run.smithi067.stderr: 63550K .......... .......... .......... .......... .......... 86% 303M 0s 2019-12-09T20:48:00.971 INFO:teuthology.orchestra.run.smithi067.stderr: 63600K .......... .......... .......... .......... .......... 86% 365M 0s 2019-12-09T20:48:00.971 INFO:teuthology.orchestra.run.smithi067.stderr: 63650K .......... .......... .......... .......... .......... 86% 360M 0s 2019-12-09T20:48:00.972 INFO:teuthology.orchestra.run.smithi067.stderr: 63700K .......... .......... .......... .......... .......... 86% 360M 0s 2019-12-09T20:48:00.972 INFO:teuthology.orchestra.run.smithi067.stderr: 63750K .......... .......... .......... .......... .......... 86% 302M 0s 2019-12-09T20:48:00.972 INFO:teuthology.orchestra.run.smithi067.stderr: 63800K .......... .......... .......... .......... .......... 86% 385M 0s 2019-12-09T20:48:00.972 INFO:teuthology.orchestra.run.smithi067.stderr: 63850K .......... .......... .......... .......... .......... 86% 406M 0s 2019-12-09T20:48:00.972 INFO:teuthology.orchestra.run.smithi067.stderr: 63900K .......... .......... .......... .......... .......... 86% 405M 0s 2019-12-09T20:48:00.972 INFO:teuthology.orchestra.run.smithi067.stderr: 63950K .......... .......... .......... .......... .......... 87% 372M 0s 2019-12-09T20:48:00.972 INFO:teuthology.orchestra.run.smithi067.stderr: 64000K .......... .......... .......... .......... .......... 87% 405M 0s 2019-12-09T20:48:00.972 INFO:teuthology.orchestra.run.smithi067.stderr: 64050K .......... .......... .......... .......... .......... 87% 414M 0s 2019-12-09T20:48:00.973 INFO:teuthology.orchestra.run.smithi067.stderr: 64100K .......... .......... .......... .......... .......... 87% 410M 0s 2019-12-09T20:48:00.973 INFO:teuthology.orchestra.run.smithi067.stderr: 64150K .......... .......... .......... .......... .......... 87% 349M 0s 2019-12-09T20:48:00.973 INFO:teuthology.orchestra.run.smithi067.stderr: 64200K .......... .......... .......... .......... .......... 87% 415M 0s 2019-12-09T20:48:00.973 INFO:teuthology.orchestra.run.smithi067.stderr: 64250K .......... .......... .......... .......... .......... 87% 28.7M 0s 2019-12-09T20:48:00.973 INFO:teuthology.orchestra.run.smithi067.stderr: 64300K .......... .......... .......... .......... .......... 87% 350M 0s 2019-12-09T20:48:00.973 INFO:teuthology.orchestra.run.smithi067.stderr: 64350K .......... .......... .......... .......... .......... 87% 325M 0s 2019-12-09T20:48:00.973 INFO:teuthology.orchestra.run.smithi067.stderr: 64400K .......... .......... .......... .......... .......... 87% 375M 0s 2019-12-09T20:48:00.974 INFO:teuthology.orchestra.run.smithi067.stderr: 64450K .......... .......... .......... .......... .......... 87% 363M 0s 2019-12-09T20:48:00.974 INFO:teuthology.orchestra.run.smithi067.stderr: 64500K .......... .......... .......... .......... .......... 87% 253M 0s 2019-12-09T20:48:00.974 INFO:teuthology.orchestra.run.smithi067.stderr: 64550K .......... .......... .......... .......... .......... 87% 257M 0s 2019-12-09T20:48:00.974 INFO:teuthology.orchestra.run.smithi067.stderr: 64600K .......... .......... .......... .......... .......... 87% 399M 0s 2019-12-09T20:48:00.974 INFO:teuthology.orchestra.run.smithi067.stderr: 64650K .......... .......... .......... .......... .......... 87% 391M 0s 2019-12-09T20:48:00.974 INFO:teuthology.orchestra.run.smithi067.stderr: 64700K .......... .......... .......... .......... .......... 88% 415M 0s 2019-12-09T20:48:00.974 INFO:teuthology.orchestra.run.smithi067.stderr: 64750K .......... .......... .......... .......... .......... 88% 64.1M 0s 2019-12-09T20:48:00.975 INFO:teuthology.orchestra.run.smithi067.stderr: 64800K .......... .......... .......... .......... .......... 88% 319M 0s 2019-12-09T20:48:00.975 INFO:teuthology.orchestra.run.smithi067.stderr: 64850K .......... .......... .......... .......... .......... 88% 316M 0s 2019-12-09T20:48:00.975 INFO:teuthology.orchestra.run.smithi067.stderr: 64900K .......... .......... .......... .......... .......... 88% 282M 0s 2019-12-09T20:48:00.975 INFO:teuthology.orchestra.run.smithi067.stderr: 64950K .......... .......... .......... .......... .......... 88% 306M 0s 2019-12-09T20:48:00.975 INFO:teuthology.orchestra.run.smithi067.stderr: 65000K .......... .......... .......... .......... .......... 88% 357M 0s 2019-12-09T20:48:00.975 INFO:teuthology.orchestra.run.smithi067.stderr: 65050K .......... .......... .......... .......... .......... 88% 358M 0s 2019-12-09T20:48:00.975 INFO:teuthology.orchestra.run.smithi067.stderr: 65100K .......... .......... .......... .......... .......... 88% 327M 0s 2019-12-09T20:48:00.976 INFO:teuthology.orchestra.run.smithi067.stderr: 65150K .......... .......... .......... .......... .......... 88% 296M 0s 2019-12-09T20:48:00.976 INFO:teuthology.orchestra.run.smithi067.stderr: 65200K .......... .......... .......... .......... .......... 88% 328M 0s 2019-12-09T20:48:00.976 INFO:teuthology.orchestra.run.smithi067.stderr: 65250K .......... .......... .......... .......... .......... 88% 333M 0s 2019-12-09T20:48:00.976 INFO:teuthology.orchestra.run.smithi067.stderr: 65300K .......... .......... .......... .......... .......... 88% 399M 0s 2019-12-09T20:48:00.976 INFO:teuthology.orchestra.run.smithi067.stderr: 65350K .......... .......... .......... .......... .......... 88% 307M 0s 2019-12-09T20:48:00.976 INFO:teuthology.orchestra.run.smithi067.stderr: 65400K .......... .......... .......... .......... .......... 88% 347M 0s 2019-12-09T20:48:00.976 INFO:teuthology.orchestra.run.smithi067.stderr: 65450K .......... .......... .......... .......... .......... 89% 354M 0s 2019-12-09T20:48:00.977 INFO:teuthology.orchestra.run.smithi067.stderr: 65500K .......... .......... .......... .......... .......... 89% 385M 0s 2019-12-09T20:48:00.977 INFO:teuthology.orchestra.run.smithi067.stderr: 65550K .......... .......... .......... .......... .......... 89% 309M 0s 2019-12-09T20:48:00.977 INFO:teuthology.orchestra.run.smithi067.stderr: 65600K .......... .......... .......... .......... .......... 89% 361M 0s 2019-12-09T20:48:00.977 INFO:teuthology.orchestra.run.smithi067.stderr: 65650K .......... .......... .......... .......... .......... 89% 395M 0s 2019-12-09T20:48:00.977 INFO:teuthology.orchestra.run.smithi067.stderr: 65700K .......... .......... .......... .......... .......... 89% 383M 0s 2019-12-09T20:48:00.977 INFO:teuthology.orchestra.run.smithi067.stderr: 65750K .......... .......... .......... .......... .......... 89% 333M 0s 2019-12-09T20:48:00.977 INFO:teuthology.orchestra.run.smithi067.stderr: 65800K .......... .......... .......... .......... .......... 89% 351M 0s 2019-12-09T20:48:00.977 INFO:teuthology.orchestra.run.smithi067.stderr: 65850K .......... .......... .......... .......... .......... 89% 390M 0s 2019-12-09T20:48:00.978 INFO:teuthology.orchestra.run.smithi067.stderr: 65900K .......... .......... .......... .......... .......... 89% 377M 0s 2019-12-09T20:48:00.978 INFO:teuthology.orchestra.run.smithi067.stderr: 65950K .......... .......... .......... .......... .......... 89% 309M 0s 2019-12-09T20:48:00.978 INFO:teuthology.orchestra.run.smithi067.stderr: 66000K .......... .......... .......... .......... .......... 89% 202M 0s 2019-12-09T20:48:00.978 INFO:teuthology.orchestra.run.smithi067.stderr: 66050K .......... .......... .......... .......... .......... 89% 377M 0s 2019-12-09T20:48:00.978 INFO:teuthology.orchestra.run.smithi067.stderr: 66100K .......... .......... .......... .......... .......... 89% 374M 0s 2019-12-09T20:48:00.978 INFO:teuthology.orchestra.run.smithi067.stderr: 66150K .......... .......... .......... .......... .......... 89% 344M 0s 2019-12-09T20:48:00.978 INFO:teuthology.orchestra.run.smithi067.stderr: 66200K .......... .......... .......... .......... .......... 90% 390M 0s 2019-12-09T20:48:00.979 INFO:teuthology.orchestra.run.smithi067.stderr: 66250K .......... .......... .......... .......... .......... 90% 392M 0s 2019-12-09T20:48:00.979 INFO:teuthology.orchestra.run.smithi067.stderr: 66300K .......... .......... .......... .......... .......... 90% 367M 0s 2019-12-09T20:48:00.979 INFO:teuthology.orchestra.run.smithi067.stderr: 66350K .......... .......... .......... .......... .......... 90% 326M 0s 2019-12-09T20:48:00.979 INFO:teuthology.orchestra.run.smithi067.stderr: 66400K .......... .......... .......... .......... .......... 90% 380M 0s 2019-12-09T20:48:00.979 INFO:teuthology.orchestra.run.smithi067.stderr: 66450K .......... .......... .......... .......... .......... 90% 395M 0s 2019-12-09T20:48:00.979 INFO:teuthology.orchestra.run.smithi067.stderr: 66500K .......... .......... .......... .......... .......... 90% 385M 0s 2019-12-09T20:48:00.979 INFO:teuthology.orchestra.run.smithi067.stderr: 66550K .......... .......... .......... .......... .......... 90% 351M 0s 2019-12-09T20:48:00.980 INFO:teuthology.orchestra.run.smithi067.stderr: 66600K .......... .......... .......... .......... .......... 90% 389M 0s 2019-12-09T20:48:00.980 INFO:teuthology.orchestra.run.smithi067.stderr: 66650K .......... .......... .......... .......... .......... 90% 380M 0s 2019-12-09T20:48:00.980 INFO:teuthology.orchestra.run.smithi067.stderr: 66700K .......... .......... .......... .......... .......... 90% 389M 0s 2019-12-09T20:48:00.980 INFO:teuthology.orchestra.run.smithi067.stderr: 66750K .......... .......... .......... .......... .......... 90% 331M 0s 2019-12-09T20:48:00.980 INFO:teuthology.orchestra.run.smithi067.stderr: 66800K .......... .......... .......... .......... .......... 90% 387M 0s 2019-12-09T20:48:00.980 INFO:teuthology.orchestra.run.smithi067.stderr: 66850K .......... .......... .......... .......... .......... 90% 376M 0s 2019-12-09T20:48:00.980 INFO:teuthology.orchestra.run.smithi067.stderr: 66900K .......... .......... .......... .......... .......... 91% 389M 0s 2019-12-09T20:48:00.981 INFO:teuthology.orchestra.run.smithi067.stderr: 66950K .......... .......... .......... .......... .......... 91% 347M 0s 2019-12-09T20:48:00.981 INFO:teuthology.orchestra.run.smithi067.stderr: 67000K .......... .......... .......... .......... .......... 91% 345M 0s 2019-12-09T20:48:00.981 INFO:teuthology.orchestra.run.smithi067.stderr: 67050K .......... .......... .......... .......... .......... 91% 319M 0s 2019-12-09T20:48:00.981 INFO:teuthology.orchestra.run.smithi067.stderr: 67100K .......... .......... .......... .......... .......... 91% 322M 0s 2019-12-09T20:48:00.981 INFO:teuthology.orchestra.run.smithi067.stderr: 67150K .......... .......... .......... .......... .......... 91% 216M 0s 2019-12-09T20:48:00.981 INFO:teuthology.orchestra.run.smithi067.stderr: 67200K .......... .......... .......... .......... .......... 91% 304M 0s 2019-12-09T20:48:00.981 INFO:teuthology.orchestra.run.smithi067.stderr: 67250K .......... .......... .......... .......... .......... 91% 349M 0s 2019-12-09T20:48:00.981 INFO:teuthology.orchestra.run.smithi067.stderr: 67300K .......... .......... .......... .......... .......... 91% 377M 0s 2019-12-09T20:48:00.982 INFO:teuthology.orchestra.run.smithi067.stderr: 67350K .......... .......... .......... .......... .......... 91% 342M 0s 2019-12-09T20:48:00.982 INFO:teuthology.orchestra.run.smithi067.stderr: 67400K .......... .......... .......... .......... .......... 91% 414M 0s 2019-12-09T20:48:00.982 INFO:teuthology.orchestra.run.smithi067.stderr: 67450K .......... .......... .......... .......... .......... 91% 404M 0s 2019-12-09T20:48:00.982 INFO:teuthology.orchestra.run.smithi067.stderr: 67500K .......... .......... .......... .......... .......... 91% 404M 0s 2019-12-09T20:48:00.982 INFO:teuthology.orchestra.run.smithi067.stderr: 67550K .......... .......... .......... .......... .......... 91% 338M 0s 2019-12-09T20:48:00.982 INFO:teuthology.orchestra.run.smithi067.stderr: 67600K .......... .......... .......... .......... .......... 91% 387M 0s 2019-12-09T20:48:00.982 INFO:teuthology.orchestra.run.smithi067.stderr: 67650K .......... .......... .......... .......... .......... 92% 386M 0s 2019-12-09T20:48:00.983 INFO:teuthology.orchestra.run.smithi067.stderr: 67700K .......... .......... .......... .......... .......... 92% 390M 0s 2019-12-09T20:48:00.983 INFO:teuthology.orchestra.run.smithi067.stderr: 67750K .......... .......... .......... .......... .......... 92% 334M 0s 2019-12-09T20:48:00.983 INFO:teuthology.orchestra.run.smithi067.stderr: 67800K .......... .......... .......... .......... .......... 92% 408M 0s 2019-12-09T20:48:00.983 INFO:teuthology.orchestra.run.smithi067.stderr: 67850K .......... .......... .......... .......... .......... 92% 355M 0s 2019-12-09T20:48:00.983 INFO:teuthology.orchestra.run.smithi067.stderr: 67900K .......... .......... .......... .......... .......... 92% 366M 0s 2019-12-09T20:48:00.983 INFO:teuthology.orchestra.run.smithi067.stderr: 67950K .......... .......... .......... .......... .......... 92% 303M 0s 2019-12-09T20:48:00.983 INFO:teuthology.orchestra.run.smithi067.stderr: 68000K .......... .......... .......... .......... .......... 92% 380M 0s 2019-12-09T20:48:00.984 INFO:teuthology.orchestra.run.smithi067.stderr: 68050K .......... .......... .......... .......... .......... 92% 358M 0s 2019-12-09T20:48:00.984 INFO:teuthology.orchestra.run.smithi067.stderr: 68100K .......... .......... .......... .......... .......... 92% 355M 0s 2019-12-09T20:48:00.984 INFO:teuthology.orchestra.run.smithi067.stderr: 68150K .......... .......... .......... .......... .......... 92% 373M 0s 2019-12-09T20:48:00.984 INFO:teuthology.orchestra.run.smithi067.stderr: 68200K .......... .......... .......... .......... .......... 92% 415M 0s 2019-12-09T20:48:00.984 INFO:teuthology.orchestra.run.smithi067.stderr: 68250K .......... .......... .......... .......... .......... 92% 78.0M 0s 2019-12-09T20:48:00.984 INFO:teuthology.orchestra.run.smithi067.stderr: 68300K .......... .......... .......... .......... .......... 92% 333M 0s 2019-12-09T20:48:00.984 INFO:teuthology.orchestra.run.smithi067.stderr: 68350K .......... .......... .......... .......... .......... 92% 266M 0s 2019-12-09T20:48:00.984 INFO:teuthology.orchestra.run.smithi067.stderr: 68400K .......... .......... .......... .......... .......... 93% 375M 0s 2019-12-09T20:48:00.985 INFO:teuthology.orchestra.run.smithi067.stderr: 68450K .......... .......... .......... .......... .......... 93% 369M 0s 2019-12-09T20:48:00.985 INFO:teuthology.orchestra.run.smithi067.stderr: 68500K .......... .......... .......... .......... .......... 93% 386M 0s 2019-12-09T20:48:00.985 INFO:teuthology.orchestra.run.smithi067.stderr: 68550K .......... .......... .......... .......... .......... 93% 346M 0s 2019-12-09T20:48:00.985 INFO:teuthology.orchestra.run.smithi067.stderr: 68600K .......... .......... .......... .......... .......... 93% 373M 0s 2019-12-09T20:48:00.985 INFO:teuthology.orchestra.run.smithi067.stderr: 68650K .......... .......... .......... .......... .......... 93% 366M 0s 2019-12-09T20:48:00.985 INFO:teuthology.orchestra.run.smithi067.stderr: 68700K .......... .......... .......... .......... .......... 93% 385M 0s 2019-12-09T20:48:00.985 INFO:teuthology.orchestra.run.smithi067.stderr: 68750K .......... .......... .......... .......... .......... 93% 317M 0s 2019-12-09T20:48:00.986 INFO:teuthology.orchestra.run.smithi067.stderr: 68800K .......... .......... .......... .......... .......... 93% 359M 0s 2019-12-09T20:48:00.986 INFO:teuthology.orchestra.run.smithi067.stderr: 68850K .......... .......... .......... .......... .......... 93% 387M 0s 2019-12-09T20:48:00.986 INFO:teuthology.orchestra.run.smithi067.stderr: 68900K .......... .......... .......... .......... .......... 93% 390M 0s 2019-12-09T20:48:00.986 INFO:teuthology.orchestra.run.smithi067.stderr: 68950K .......... .......... .......... .......... .......... 93% 343M 0s 2019-12-09T20:48:00.986 INFO:teuthology.orchestra.run.smithi067.stderr: 69000K .......... .......... .......... .......... .......... 93% 347M 0s 2019-12-09T20:48:00.986 INFO:teuthology.orchestra.run.smithi067.stderr: 69050K .......... .......... .......... .......... .......... 93% 299M 0s 2019-12-09T20:48:00.986 INFO:teuthology.orchestra.run.smithi067.stderr: 69100K .......... .......... .......... .......... .......... 94% 362M 0s 2019-12-09T20:48:00.987 INFO:teuthology.orchestra.run.smithi067.stderr: 69150K .......... .......... .......... .......... .......... 94% 313M 0s 2019-12-09T20:48:00.987 INFO:teuthology.orchestra.run.smithi067.stderr: 69200K .......... .......... .......... .......... .......... 94% 381M 0s 2019-12-09T20:48:00.987 INFO:teuthology.orchestra.run.smithi067.stderr: 69250K .......... .......... .......... .......... .......... 94% 373M 0s 2019-12-09T20:48:00.987 INFO:teuthology.orchestra.run.smithi067.stderr: 69300K .......... .......... .......... .......... .......... 94% 364M 0s 2019-12-09T20:48:00.987 INFO:teuthology.orchestra.run.smithi067.stderr: 69350K .......... .......... .......... .......... .......... 94% 314M 0s 2019-12-09T20:48:00.987 INFO:teuthology.orchestra.run.smithi067.stderr: 69400K .......... .......... .......... .......... .......... 94% 350M 0s 2019-12-09T20:48:00.987 INFO:teuthology.orchestra.run.smithi067.stderr: 69450K .......... .......... .......... .......... .......... 94% 354M 0s 2019-12-09T20:48:00.987 INFO:teuthology.orchestra.run.smithi067.stderr: 69500K .......... .......... .......... .......... .......... 94% 381M 0s 2019-12-09T20:48:00.988 INFO:teuthology.orchestra.run.smithi067.stderr: 69550K .......... .......... .......... .......... .......... 94% 317M 0s 2019-12-09T20:48:00.988 INFO:teuthology.orchestra.run.smithi067.stderr: 69600K .......... .......... .......... .......... .......... 94% 379M 0s 2019-12-09T20:48:00.988 INFO:teuthology.orchestra.run.smithi067.stderr: 69650K .......... .......... .......... .......... .......... 94% 349M 0s 2019-12-09T20:48:00.988 INFO:teuthology.orchestra.run.smithi067.stderr: 69700K .......... .......... .......... .......... .......... 94% 379M 0s 2019-12-09T20:48:00.988 INFO:teuthology.orchestra.run.smithi067.stderr: 69750K .......... .......... .......... .......... .......... 94% 331M 0s 2019-12-09T20:48:00.988 INFO:teuthology.orchestra.run.smithi067.stderr: 69800K .......... .......... .......... .......... .......... 94% 314M 0s 2019-12-09T20:48:00.988 INFO:teuthology.orchestra.run.smithi067.stderr: 69850K .......... .......... .......... .......... .......... 95% 325M 0s 2019-12-09T20:48:00.989 INFO:teuthology.orchestra.run.smithi067.stderr: 69900K .......... .......... .......... .......... .......... 95% 318M 0s 2019-12-09T20:48:00.989 INFO:teuthology.orchestra.run.smithi067.stderr: 69950K .......... .......... .......... .......... .......... 95% 297M 0s 2019-12-09T20:48:00.989 INFO:teuthology.orchestra.run.smithi067.stderr: 70000K .......... .......... .......... .......... .......... 95% 368M 0s 2019-12-09T20:48:00.989 INFO:teuthology.orchestra.run.smithi067.stderr: 70050K .......... .......... .......... .......... .......... 95% 377M 0s 2019-12-09T20:48:00.989 INFO:teuthology.orchestra.run.smithi067.stderr: 70100K .......... .......... .......... .......... .......... 95% 367M 0s 2019-12-09T20:48:00.989 INFO:teuthology.orchestra.run.smithi067.stderr: 70150K .......... .......... .......... .......... .......... 95% 324M 0s 2019-12-09T20:48:00.989 INFO:teuthology.orchestra.run.smithi067.stderr: 70200K .......... .......... .......... .......... .......... 95% 350M 0s 2019-12-09T20:48:00.990 INFO:teuthology.orchestra.run.smithi067.stderr: 70250K .......... .......... .......... .......... .......... 95% 370M 0s 2019-12-09T20:48:00.990 INFO:teuthology.orchestra.run.smithi067.stderr: 70300K .......... .......... .......... .......... .......... 95% 371M 0s 2019-12-09T20:48:00.990 INFO:teuthology.orchestra.run.smithi067.stderr: 70350K .......... .......... .......... .......... .......... 95% 276M 0s 2019-12-09T20:48:00.990 INFO:teuthology.orchestra.run.smithi067.stderr: 70400K .......... .......... .......... .......... .......... 95% 321M 0s 2019-12-09T20:48:00.990 INFO:teuthology.orchestra.run.smithi067.stderr: 70450K .......... .......... .......... .......... .......... 95% 382M 0s 2019-12-09T20:48:00.990 INFO:teuthology.orchestra.run.smithi067.stderr: 70500K .......... .......... .......... .......... .......... 95% 356M 0s 2019-12-09T20:48:00.990 INFO:teuthology.orchestra.run.smithi067.stderr: 70550K .......... .......... .......... .......... .......... 95% 298M 0s 2019-12-09T20:48:00.991 INFO:teuthology.orchestra.run.smithi067.stderr: 70600K .......... .......... .......... .......... .......... 96% 373M 0s 2019-12-09T20:48:00.991 INFO:teuthology.orchestra.run.smithi067.stderr: 70650K .......... .......... .......... .......... .......... 96% 375M 0s 2019-12-09T20:48:00.991 INFO:teuthology.orchestra.run.smithi067.stderr: 70700K .......... .......... .......... .......... .......... 96% 374M 0s 2019-12-09T20:48:00.991 INFO:teuthology.orchestra.run.smithi067.stderr: 70750K .......... .......... .......... .......... .......... 96% 327M 0s 2019-12-09T20:48:00.991 INFO:teuthology.orchestra.run.smithi067.stderr: 70800K .......... .......... .......... .......... .......... 96% 345M 0s 2019-12-09T20:48:00.991 INFO:teuthology.orchestra.run.smithi067.stderr: 70850K .......... .......... .......... .......... .......... 96% 405M 0s 2019-12-09T20:48:00.991 INFO:teuthology.orchestra.run.smithi067.stderr: 70900K .......... .......... .......... .......... .......... 96% 396M 0s 2019-12-09T20:48:00.991 INFO:teuthology.orchestra.run.smithi067.stderr: 70950K .......... .......... .......... .......... .......... 96% 350M 0s 2019-12-09T20:48:00.992 INFO:teuthology.orchestra.run.smithi067.stderr: 71000K .......... .......... .......... .......... .......... 96% 390M 0s 2019-12-09T20:48:00.992 INFO:teuthology.orchestra.run.smithi067.stderr: 71050K .......... .......... .......... .......... .......... 96% 404M 0s 2019-12-09T20:48:00.992 INFO:teuthology.orchestra.run.smithi067.stderr: 71100K .......... .......... .......... .......... .......... 96% 394M 0s 2019-12-09T20:48:00.992 INFO:teuthology.orchestra.run.smithi067.stderr: 71150K .......... .......... .......... .......... .......... 96% 316M 0s 2019-12-09T20:48:00.992 INFO:teuthology.orchestra.run.smithi067.stderr: 71200K .......... .......... .......... .......... .......... 96% 387M 0s 2019-12-09T20:48:00.992 INFO:teuthology.orchestra.run.smithi067.stderr: 71250K .......... .......... .......... .......... .......... 96% 389M 0s 2019-12-09T20:48:00.992 INFO:teuthology.orchestra.run.smithi067.stderr: 71300K .......... .......... .......... .......... .......... 96% 376M 0s 2019-12-09T20:48:00.993 INFO:teuthology.orchestra.run.smithi067.stderr: 71350K .......... .......... .......... .......... .......... 97% 360M 0s 2019-12-09T20:48:00.993 INFO:teuthology.orchestra.run.smithi067.stderr: 71400K .......... .......... .......... .......... .......... 97% 389M 0s 2019-12-09T20:48:00.993 INFO:teuthology.orchestra.run.smithi067.stderr: 71450K .......... .......... .......... .......... .......... 97% 391M 0s 2019-12-09T20:48:00.993 INFO:teuthology.orchestra.run.smithi067.stderr: 71500K .......... .......... .......... .......... .......... 97% 299M 0s 2019-12-09T20:48:00.993 INFO:teuthology.orchestra.run.smithi067.stderr: 71550K .......... .......... .......... .......... .......... 97% 309M 0s 2019-12-09T20:48:00.993 INFO:teuthology.orchestra.run.smithi067.stderr: 71600K .......... .......... .......... .......... .......... 97% 389M 0s 2019-12-09T20:48:00.993 INFO:teuthology.orchestra.run.smithi067.stderr: 71650K .......... .......... .......... .......... .......... 97% 385M 0s 2019-12-09T20:48:00.994 INFO:teuthology.orchestra.run.smithi067.stderr: 71700K .......... .......... .......... .......... .......... 97% 390M 0s 2019-12-09T20:48:00.994 INFO:teuthology.orchestra.run.smithi067.stderr: 71750K .......... .......... .......... .......... .......... 97% 341M 0s 2019-12-09T20:48:00.994 INFO:teuthology.orchestra.run.smithi067.stderr: 71800K .......... .......... .......... .......... .......... 97% 391M 0s 2019-12-09T20:48:00.994 INFO:teuthology.orchestra.run.smithi067.stderr: 71850K .......... .......... .......... .......... .......... 97% 406M 0s 2019-12-09T20:48:00.994 INFO:teuthology.orchestra.run.smithi067.stderr: 71900K .......... .......... .......... .......... .......... 97% 391M 0s 2019-12-09T20:48:00.994 INFO:teuthology.orchestra.run.smithi067.stderr: 71950K .......... .......... .......... .......... .......... 97% 329M 0s 2019-12-09T20:48:00.994 INFO:teuthology.orchestra.run.smithi067.stderr: 72000K .......... .......... .......... .......... .......... 97% 376M 0s 2019-12-09T20:48:00.994 INFO:teuthology.orchestra.run.smithi067.stderr: 72050K .......... .......... .......... .......... .......... 98% 380M 0s 2019-12-09T20:48:00.995 INFO:teuthology.orchestra.run.smithi067.stderr: 72100K .......... .......... .......... .......... .......... 98% 391M 0s 2019-12-09T20:48:00.995 INFO:teuthology.orchestra.run.smithi067.stderr: 72150K .......... .......... .......... .......... .......... 98% 345M 0s 2019-12-09T20:48:00.995 INFO:teuthology.orchestra.run.smithi067.stderr: 72200K .......... .......... .......... .......... .......... 98% 396M 0s 2019-12-09T20:48:00.995 INFO:teuthology.orchestra.run.smithi067.stderr: 72250K .......... .......... .......... .......... .......... 98% 408M 0s 2019-12-09T20:48:00.995 INFO:teuthology.orchestra.run.smithi067.stderr: 72300K .......... .......... .......... .......... .......... 98% 396M 0s 2019-12-09T20:48:00.995 INFO:teuthology.orchestra.run.smithi067.stderr: 72350K .......... .......... .......... .......... .......... 98% 345M 0s 2019-12-09T20:48:00.995 INFO:teuthology.orchestra.run.smithi067.stderr: 72400K .......... .......... .......... .......... .......... 98% 374M 0s 2019-12-09T20:48:00.996 INFO:teuthology.orchestra.run.smithi067.stderr: 72450K .......... .......... .......... .......... .......... 98% 391M 0s 2019-12-09T20:48:00.996 INFO:teuthology.orchestra.run.smithi067.stderr: 72500K .......... .......... .......... .......... .......... 98% 393M 0s 2019-12-09T20:48:00.996 INFO:teuthology.orchestra.run.smithi067.stderr: 72550K .......... .......... .......... .......... .......... 98% 344M 0s 2019-12-09T20:48:00.996 INFO:teuthology.orchestra.run.smithi067.stderr: 72600K .......... .......... .......... .......... .......... 98% 376M 0s 2019-12-09T20:48:00.996 INFO:teuthology.orchestra.run.smithi067.stderr: 72650K .......... .......... .......... .......... .......... 98% 399M 0s 2019-12-09T20:48:00.996 INFO:teuthology.orchestra.run.smithi067.stderr: 72700K .......... .......... .......... .......... .......... 98% 375M 0s 2019-12-09T20:48:00.996 INFO:teuthology.orchestra.run.smithi067.stderr: 72750K .......... .......... .......... .......... .......... 98% 322M 0s 2019-12-09T20:48:00.997 INFO:teuthology.orchestra.run.smithi067.stderr: 72800K .......... .......... .......... .......... .......... 99% 397M 0s 2019-12-09T20:48:00.997 INFO:teuthology.orchestra.run.smithi067.stderr: 72850K .......... .......... .......... .......... .......... 99% 398M 0s 2019-12-09T20:48:00.997 INFO:teuthology.orchestra.run.smithi067.stderr: 72900K .......... .......... .......... .......... .......... 99% 403M 0s 2019-12-09T20:48:00.997 INFO:teuthology.orchestra.run.smithi067.stderr: 72950K .......... .......... .......... .......... .......... 99% 345M 0s 2019-12-09T20:48:00.997 INFO:teuthology.orchestra.run.smithi067.stderr: 73000K .......... .......... .......... .......... .......... 99% 340M 0s 2019-12-09T20:48:00.997 INFO:teuthology.orchestra.run.smithi067.stderr: 73050K .......... .......... .......... .......... .......... 99% 395M 0s 2019-12-09T20:48:00.997 INFO:teuthology.orchestra.run.smithi067.stderr: 73100K .......... .......... .......... .......... .......... 99% 411M 0s 2019-12-09T20:48:00.997 INFO:teuthology.orchestra.run.smithi067.stderr: 73150K .......... .......... .......... .......... .......... 99% 307M 0s 2019-12-09T20:48:00.998 INFO:teuthology.orchestra.run.smithi067.stderr: 73200K .......... .......... .......... .......... .......... 99% 356M 0s 2019-12-09T20:48:00.998 INFO:teuthology.orchestra.run.smithi067.stderr: 73250K .......... .......... .......... .......... .......... 99% 325M 0s 2019-12-09T20:48:00.998 INFO:teuthology.orchestra.run.smithi067.stderr: 73300K .......... .......... .......... .......... .......... 99% 357M 0s 2019-12-09T20:48:00.998 INFO:teuthology.orchestra.run.smithi067.stderr: 73350K .......... .......... .......... .......... .......... 99% 325M 0s 2019-12-09T20:48:00.998 INFO:teuthology.orchestra.run.smithi067.stderr: 73400K .......... .......... .......... .......... .......... 99% 408M 0s 2019-12-09T20:48:00.998 INFO:teuthology.orchestra.run.smithi067.stderr: 73450K .......... .......... .......... .......... .......... 99% 387M 0s 2019-12-09T20:48:00.998 INFO:teuthology.orchestra.run.smithi067.stderr: 73500K .......... .......... .......... .......... .......... 99% 415M 0s 2019-12-09T20:48:00.999 INFO:teuthology.orchestra.run.smithi067.stderr: 73550K ........ 100% 372M=0.4s 2019-12-09T20:48:00.999 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T20:48:00.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09 20:47:59 (183 MB/s) - ‘gradle-4.7-bin.zip’ saved [75323957/75323957] 2019-12-09T20:48:00.999 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T20:48:00.999 INFO:teuthology.orchestra.run.smithi067.stdout:Archive: gradle-4.7-bin.zip 2019-12-09T20:48:00.999 INFO:teuthology.orchestra.run.smithi067.stdout: creating: /opt/gradle/gradle-4.7/ 2019-12-09T20:48:01.000 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/getting-started.html 2019-12-09T20:48:01.000 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/LICENSE 2019-12-09T20:48:01.000 INFO:teuthology.orchestra.run.smithi067.stdout: creating: /opt/gradle/gradle-4.7/media/ 2019-12-09T20:48:01.000 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-64x64.png 2019-12-09T20:48:01.000 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-48x48.png 2019-12-09T20:48:01.000 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle.icns 2019-12-09T20:48:01.000 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-24x24.png 2019-12-09T20:48:01.001 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-256x256.png 2019-12-09T20:48:01.001 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-512x512.png 2019-12-09T20:48:01.001 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-16x16.png 2019-12-09T20:48:01.001 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-32x32.png 2019-12-09T20:48:01.001 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/media/gradle-icon-128x128.png 2019-12-09T20:48:01.001 INFO:teuthology.orchestra.run.smithi067.stdout: creating: /opt/gradle/gradle-4.7/init.d/ 2019-12-09T20:48:01.001 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/init.d/readme.txt 2019-12-09T20:48:01.001 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/NOTICE 2019-12-09T20:48:01.002 INFO:teuthology.orchestra.run.smithi067.stdout: creating: /opt/gradle/gradle-4.7/bin/ 2019-12-09T20:48:01.002 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/bin/gradle.bat 2019-12-09T20:48:01.002 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/bin/gradle 2019-12-09T20:48:01.002 INFO:teuthology.orchestra.run.smithi067.stdout: creating: /opt/gradle/gradle-4.7/lib/ 2019-12-09T20:48:01.002 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/annotations-13.0.jar 2019-12-09T20:48:01.002 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-kotlin-dsl-0.16.3.jar 2019-12-09T20:48:01.002 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-kotlin-dsl-tooling-builders-0.16.3.jar 2019-12-09T20:48:01.003 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-kotlin-dsl-tooling-models-0.16.3.jar 2019-12-09T20:48:01.003 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-compiler-embeddable-1.2.31.jar 2019-12-09T20:48:01.003 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-reflect-1.2.31.jar 2019-12-09T20:48:01.003 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-sam-with-receiver-compiler-plugin-1.2.31.jar 2019-12-09T20:48:01.003 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-script-runtime-1.2.31.jar 2019-12-09T20:48:01.003 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-stdlib-1.2.31.jar 2019-12-09T20:48:01.003 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-stdlib-jdk7-1.2.31.jar 2019-12-09T20:48:01.004 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/kotlin-stdlib-jdk8-1.2.31.jar 2019-12-09T20:48:01.004 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-launcher-4.7.jar 2019-12-09T20:48:01.004 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-runtime-api-info-4.7.jar 2019-12-09T20:48:01.004 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-tooling-api-4.7.jar 2019-12-09T20:48:01.004 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-wrapper-4.7.jar 2019-12-09T20:48:01.004 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-installation-beacon-4.7.jar 2019-12-09T20:48:01.004 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-core-4.7.jar 2019-12-09T20:48:01.005 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-jvm-services-4.7.jar 2019-12-09T20:48:01.005 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-core-api-4.7.jar 2019-12-09T20:48:01.005 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-build-cache-4.7.jar 2019-12-09T20:48:01.005 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-logging-4.7.jar 2019-12-09T20:48:01.005 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-process-services-4.7.jar 2019-12-09T20:48:01.005 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-resources-4.7.jar 2019-12-09T20:48:01.005 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-persistent-cache-4.7.jar 2019-12-09T20:48:01.006 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-native-4.7.jar 2019-12-09T20:48:01.006 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-model-groovy-4.7.jar 2019-12-09T20:48:01.006 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-model-core-4.7.jar 2019-12-09T20:48:01.006 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-base-services-groovy-4.7.jar 2019-12-09T20:48:01.006 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-messaging-4.7.jar 2019-12-09T20:48:01.006 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-base-services-4.7.jar 2019-12-09T20:48:01.006 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-build-option-4.7.jar 2019-12-09T20:48:01.006 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-cli-4.7.jar 2019-12-09T20:48:01.007 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/gradle-docs-4.7.jar 2019-12-09T20:48:01.007 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/asm-commons-6.1.1.jar 2019-12-09T20:48:01.007 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/asm-analysis-6.1.1.jar 2019-12-09T20:48:01.007 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/asm-tree-6.1.1.jar 2019-12-09T20:48:01.007 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/kryo-2.20.jar 2019-12-09T20:48:01.007 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/reflectasm-1.07-shaded.jar 2019-12-09T20:48:01.007 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/asm-6.1.1.jar 2019-12-09T20:48:01.008 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/commons-io-2.4.jar 2019-12-09T20:48:01.008 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/jcl-over-slf4j-1.7.16.jar 2019-12-09T20:48:01.008 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/jul-to-slf4j-1.7.16.jar 2019-12-09T20:48:01.008 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/log4j-over-slf4j-1.7.16.jar 2019-12-09T20:48:01.008 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/slf4j-api-1.7.16.jar 2019-12-09T20:48:01.008 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/guava-jdk5-17.0.jar 2019-12-09T20:48:01.008 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/jsr305-2.0.1.jar 2019-12-09T20:48:01.009 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/fastutil-7.2.1-min.jar 2019-12-09T20:48:01.009 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/commons-lang-2.6.jar 2019-12-09T20:48:01.009 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/jcip-annotations-1.0.jar 2019-12-09T20:48:01.009 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/groovy-all-2.4.12.jar 2019-12-09T20:48:01.009 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/ant-1.9.9.jar 2019-12-09T20:48:01.009 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/javax.inject-1.jar 2019-12-09T20:48:01.009 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/commons-collections-3.2.2.jar 2019-12-09T20:48:01.010 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-0.14.jar 2019-12-09T20:48:01.010 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/commons-compress-1.14.jar 2019-12-09T20:48:01.010 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/xml-apis-1.4.01.jar 2019-12-09T20:48:01.010 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/jansi-1.14.jar 2019-12-09T20:48:01.010 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/ant-launcher-1.9.9.jar 2019-12-09T20:48:01.010 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-freebsd-amd64-libcpp-0.14.jar 2019-12-09T20:48:01.010 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-freebsd-amd64-libstdcpp-0.14.jar 2019-12-09T20:48:01.011 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-freebsd-i386-libcpp-0.14.jar 2019-12-09T20:48:01.011 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-freebsd-i386-libstdcpp-0.14.jar 2019-12-09T20:48:01.011 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-amd64-0.14.jar 2019-12-09T20:48:01.011 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-i386-0.14.jar 2019-12-09T20:48:01.011 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-osx-amd64-0.14.jar 2019-12-09T20:48:01.011 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-osx-i386-0.14.jar 2019-12-09T20:48:01.011 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-windows-amd64-0.14.jar 2019-12-09T20:48:01.012 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-windows-i386-0.14.jar 2019-12-09T20:48:01.012 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-amd64-ncurses5-0.14.jar 2019-12-09T20:48:01.012 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-amd64-ncurses6-0.14.jar 2019-12-09T20:48:01.012 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-i386-ncurses5-0.14.jar 2019-12-09T20:48:01.012 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/native-platform-linux-i386-ncurses6-0.14.jar 2019-12-09T20:48:01.012 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/minlog-1.2.jar 2019-12-09T20:48:01.012 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/objenesis-1.2.jar 2019-12-09T20:48:01.013 INFO:teuthology.orchestra.run.smithi067.stdout: creating: /opt/gradle/gradle-4.7/lib/plugins/ 2019-12-09T20:48:01.013 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-announce-4.7.jar 2019-12-09T20:48:01.013 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-antlr-4.7.jar 2019-12-09T20:48:01.013 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-code-quality-4.7.jar 2019-12-09T20:48:01.013 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-osgi-4.7.jar 2019-12-09T20:48:01.013 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-signing-4.7.jar 2019-12-09T20:48:01.013 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ide-native-4.7.jar 2019-12-09T20:48:01.013 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-testing-native-4.7.jar 2019-12-09T20:48:01.014 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-native-4.7.jar 2019-12-09T20:48:01.014 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-plugin-development-4.7.jar 2019-12-09T20:48:01.014 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-maven-4.7.jar 2019-12-09T20:48:01.014 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ide-play-4.7.jar 2019-12-09T20:48:01.014 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ide-4.7.jar 2019-12-09T20:48:01.014 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-scala-4.7.jar 2019-12-09T20:48:01.014 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-build-comparison-4.7.jar 2019-12-09T20:48:01.015 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ear-4.7.jar 2019-12-09T20:48:01.015 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-platform-play-4.7.jar 2019-12-09T20:48:01.015 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-javascript-4.7.jar 2019-12-09T20:48:01.015 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-ivy-4.7.jar 2019-12-09T20:48:01.015 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-jacoco-4.7.jar 2019-12-09T20:48:01.015 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-build-init-4.7.jar 2019-12-09T20:48:01.015 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-plugins-4.7.jar 2019-12-09T20:48:01.016 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-tooling-api-builders-4.7.jar 2019-12-09T20:48:01.016 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-test-kit-4.7.jar 2019-12-09T20:48:01.016 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-composite-builds-4.7.jar 2019-12-09T20:48:01.016 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-groovy-4.7.jar 2019-12-09T20:48:01.016 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-scala-4.7.jar 2019-12-09T20:48:01.016 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-testing-junit-platform-4.7.jar 2019-12-09T20:48:01.016 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-testing-jvm-4.7.jar 2019-12-09T20:48:01.017 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-java-4.7.jar 2019-12-09T20:48:01.017 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-language-jvm-4.7.jar 2019-12-09T20:48:01.017 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-platform-jvm-4.7.jar 2019-12-09T20:48:01.017 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-platform-native-4.7.jar 2019-12-09T20:48:01.017 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-diagnostics-4.7.jar 2019-12-09T20:48:01.017 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-testing-base-4.7.jar 2019-12-09T20:48:01.017 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-reporting-4.7.jar 2019-12-09T20:48:01.018 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-publish-4.7.jar 2019-12-09T20:48:01.018 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-platform-base-4.7.jar 2019-12-09T20:48:01.018 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-plugin-use-4.7.jar 2019-12-09T20:48:01.018 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-resources-s3-4.7.jar 2019-12-09T20:48:01.018 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-resources-gcs-4.7.jar 2019-12-09T20:48:01.018 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-build-cache-http-4.7.jar 2019-12-09T20:48:01.018 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-dependency-management-4.7.jar 2019-12-09T20:48:01.018 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-resources-http-4.7.jar 2019-12-09T20:48:01.019 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-resources-sftp-4.7.jar 2019-12-09T20:48:01.019 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-workers-4.7.jar 2019-12-09T20:48:01.019 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gradle-version-control-4.7.jar 2019-12-09T20:48:01.019 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aws-java-sdk-s3-1.11.267.jar 2019-12-09T20:48:01.019 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aws-java-sdk-kms-1.11.267.jar 2019-12-09T20:48:01.019 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aws-java-sdk-core-1.11.267.jar 2019-12-09T20:48:01.019 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/org.eclipse.jgit-4.5.3.201708160445-r.jar 2019-12-09T20:48:01.020 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/wagon-http-2.4.jar 2019-12-09T20:48:01.020 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/wagon-http-shared4-2.4.jar 2019-12-09T20:48:01.020 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-api-services-storage-v1-rev116-1.23.0.jar 2019-12-09T20:48:01.020 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-api-client-1.23.0.jar 2019-12-09T20:48:01.020 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-oauth-client-1.23.0.jar 2019-12-09T20:48:01.020 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-http-client-jackson2-1.23.0.jar 2019-12-09T20:48:01.020 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/google-http-client-1.23.0.jar 2019-12-09T20:48:01.021 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/httpclient-4.5.5.jar 2019-12-09T20:48:01.021 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/commons-codec-1.10.jar 2019-12-09T20:48:01.021 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-container-default-1.5.5.jar 2019-12-09T20:48:01.021 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/xbean-reflect-3.4.jar 2019-12-09T20:48:01.021 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/asm-util-6.1.1.jar 2019-12-09T20:48:01.021 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jatl-0.2.2.jar 2019-12-09T20:48:01.021 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/commons-cli-1.2.jar 2019-12-09T20:48:01.022 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/biz.aQute.bndlib-3.4.0.jar 2019-12-09T20:48:01.022 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/ivy-2.2.0.jar 2019-12-09T20:48:01.022 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-compat-3.0.4.jar 2019-12-09T20:48:01.022 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-core-3.0.4.jar 2019-12-09T20:48:01.022 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/pmaven-common-0.8-20100325.jar 2019-12-09T20:48:01.022 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/pmaven-groovy-0.8-20100325.jar 2019-12-09T20:48:01.022 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/wagon-file-2.4.jar 2019-12-09T20:48:01.023 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-connector-wagon-1.13.1.jar 2019-12-09T20:48:01.023 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/bcpg-jdk15on-1.58.jar 2019-12-09T20:48:01.023 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/rhino-1.7R3.jar 2019-12-09T20:48:01.023 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/gson-2.7.jar 2019-12-09T20:48:01.023 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/simple-4.1.21.jar 2019-12-09T20:48:01.023 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-plugin-api-3.0.4.jar 2019-12-09T20:48:01.023 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/testng-6.3.1.jar 2019-12-09T20:48:01.023 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/snakeyaml-1.6.jar 2019-12-09T20:48:01.024 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/dd-plist-1.20.jar 2019-12-09T20:48:01.024 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/junit-4.12.jar 2019-12-09T20:48:01.024 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/bsh-2.0b6.jar 2019-12-09T20:48:01.024 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/junit-platform-launcher-1.0.3.jar 2019-12-09T20:48:01.024 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jcifs-1.3.17.jar 2019-12-09T20:48:01.024 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/nekohtml-1.9.20.jar 2019-12-09T20:48:01.024 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/xercesImpl-2.11.0.jar 2019-12-09T20:48:01.025 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jsch-0.1.54.jar 2019-12-09T20:48:01.025 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jmespath-java-1.11.267.jar 2019-12-09T20:48:01.025 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jackson-databind-2.8.11.jar 2019-12-09T20:48:01.025 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jackson-core-2.8.11.jar 2019-12-09T20:48:01.025 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jackson-annotations-2.8.11.jar 2019-12-09T20:48:01.025 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/joda-time-2.8.2.jar 2019-12-09T20:48:01.025 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/bcprov-jdk15on-1.58.jar 2019-12-09T20:48:01.026 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-aether-provider-3.0.4.jar 2019-12-09T20:48:01.026 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-impl-1.13.1.jar 2019-12-09T20:48:01.026 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-spi-1.13.1.jar 2019-12-09T20:48:01.026 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-util-1.13.1.jar 2019-12-09T20:48:01.026 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/aether-api-1.13.1.jar 2019-12-09T20:48:01.026 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/wagon-provider-api-2.4.jar 2019-12-09T20:48:01.026 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-classworlds-2.4.jar 2019-12-09T20:48:01.027 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-model-builder-3.0.4.jar 2019-12-09T20:48:01.027 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-model-3.0.4.jar 2019-12-09T20:48:01.027 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-settings-builder-3.0.4.jar 2019-12-09T20:48:01.027 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-artifact-3.0.4.jar 2019-12-09T20:48:01.027 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-settings-3.0.4.jar 2019-12-09T20:48:01.027 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-sec-dispatcher-1.3.jar 2019-12-09T20:48:01.027 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/maven-repository-metadata-3.0.4.jar 2019-12-09T20:48:01.027 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-utils-3.0.8.jar 2019-12-09T20:48:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/ion-java-1.0.2.jar 2019-12-09T20:48:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/httpcore-4.4.9.jar 2019-12-09T20:48:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/hamcrest-core-1.3.jar 2019-12-09T20:48:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/junit-platform-engine-1.0.3.jar 2019-12-09T20:48:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/junit-platform-commons-1.0.3.jar 2019-12-09T20:48:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/apiguardian-api-1.0.0.jar 2019-12-09T20:48:01.028 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/jcommander-1.47.jar 2019-12-09T20:48:01.029 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-interpolation-1.14.jar 2019-12-09T20:48:01.029 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-component-annotations-1.5.5.jar 2019-12-09T20:48:01.029 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/opentest4j-1.0.0.jar 2019-12-09T20:48:01.029 INFO:teuthology.orchestra.run.smithi067.stdout: inflating: /opt/gradle/gradle-4.7/lib/plugins/plexus-cipher-1.7.jar 2019-12-09T20:48:01.029 INFO:teuthology.orchestra.run.smithi067.stdout:export PATH=/opt/gradle/gradle-4.7/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin 2019-12-09T20:48:01.626 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:01.626 INFO:teuthology.orchestra.run.smithi067.stdout:Welcome to Gradle 4.7! 2019-12-09T20:48:01.626 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:01.626 INFO:teuthology.orchestra.run.smithi067.stdout:Here are the highlights of this release: 2019-12-09T20:48:01.627 INFO:teuthology.orchestra.run.smithi067.stdout: - Incremental annotation processing 2019-12-09T20:48:01.627 INFO:teuthology.orchestra.run.smithi067.stdout: - JDK 10 support 2019-12-09T20:48:01.627 INFO:teuthology.orchestra.run.smithi067.stdout: - Grouped non-interactive console logs 2019-12-09T20:48:01.627 INFO:teuthology.orchestra.run.smithi067.stdout: - Failed tests are re-run first for quicker feedback 2019-12-09T20:48:01.627 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:01.628 INFO:teuthology.orchestra.run.smithi067.stdout:For more details see https://docs.gradle.org/4.7/release-notes.html 2019-12-09T20:48:01.628 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:02.002 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:02.002 INFO:teuthology.orchestra.run.smithi067.stdout:------------------------------------------------------------ 2019-12-09T20:48:02.002 INFO:teuthology.orchestra.run.smithi067.stdout:Gradle 4.7 2019-12-09T20:48:02.003 INFO:teuthology.orchestra.run.smithi067.stdout:------------------------------------------------------------ 2019-12-09T20:48:02.003 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:02.003 INFO:teuthology.orchestra.run.smithi067.stdout:Build time: 2018-04-18 09:09:12 UTC 2019-12-09T20:48:02.003 INFO:teuthology.orchestra.run.smithi067.stdout:Revision: b9a962bf70638332300e7f810689cb2febbd4a6c 2019-12-09T20:48:02.003 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:02.004 INFO:teuthology.orchestra.run.smithi067.stdout:Groovy: 2.4.12 2019-12-09T20:48:02.004 INFO:teuthology.orchestra.run.smithi067.stdout:Ant: Apache Ant(TM) version 1.9.9 compiled on February 2 2017 2019-12-09T20:48:02.004 INFO:teuthology.orchestra.run.smithi067.stdout:JVM: 1.8.0_232 (Oracle Corporation 25.232-b09) 2019-12-09T20:48:02.004 INFO:teuthology.orchestra.run.smithi067.stdout:OS: Linux 3.10.0-1062.9.1.el7.x86_64 amd64 2019-12-09T20:48:02.004 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:02.007 DEBUG:tasks.s3tests_java:S3 Tests Java: BEGIN 2019-12-09T20:48:02.007 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2019-12-09T20:48:02.007 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi081.front.sepia.ceph.com 2019-12-09T20:48:02.008 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-12-09T20:48:02.008 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi067.front.sepia.ceph.com 2019-12-09T20:48:02.008 INFO:tasks.s3tests_java:S3 Tests Java: Creating S3 users... 2019-12-09T20:48:02.385 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf is {'bucket_prefix': 'test-', 's3main': {'access_key': None, 's3dir': 'folder', 'endpoint': None, 'display_name': None, 'region': 'mexico', 'port': None, 'access_secret': None, 'SSE': 'AES256', 'is_secure': None, 'email': None, 'dir': '../data'}} 2019-12-09T20:48:02.386 DEBUG:tasks.s3tests_java:S3 Tests Java: Creating user tester.client.0 2019-12-09T20:48:02.386 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf[s3main] is {'access_key': 'IABVLZVYEVPWADCATUBE', 's3dir': 'folder', 'endpoint': 'smithi067.front.sepia.ceph.com:80', 'display_name': 'Ms. tester.client.0', 'user_id': 'tester.client.0', 'region': 'mexico', 'port': 80, 'access_secret': 'LBcA9UK2S2vidLDphkMHkTuUZwrsTLKA9xp4Zng+rvCl5S+WNZSq2g==', 'host': 'smithi067.front.sepia.ceph.com', 'SSE': 'AES256', 'is_secure': False, 'email': 'tester.client.0_test@test.test', 'dir': '../data'} 2019-12-09T20:48:02.386 DEBUG:tasks.s3tests_java:S3 Tests Java: Setion, User = s3main, tester.client.0 2019-12-09T20:48:02.386 INFO:tasks.s3tests_java:['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '-n', 'client.0', 'user', 'create', '--uid', 'tester.client.0', '--display-name', 'Ms. tester.client.0', '--access-key', 'IABVLZVYEVPWADCATUBE', '--secret', 'LBcA9UK2S2vidLDphkMHkTuUZwrsTLKA9xp4Zng+rvCl5S+WNZSq2g==', '--email', 'tester.client.0_test@test.test', '--cluster', 'ceph'] 2019-12-09T20:48:02.387 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid tester.client.0 --display-name 'Ms. tester.client.0' --access-key IABVLZVYEVPWADCATUBE --secret LBcA9UK2S2vidLDphkMHkTuUZwrsTLKA9xp4Zng+rvCl5S+WNZSq2g== --email tester.client.0_test@test.test --cluster ceph 2019-12-09T20:48:02.521 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.455+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.558 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.492+0000 7f207043dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:02.558 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.492+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.506+0000 7f207043dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:02.571 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.506+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.512+0000 7f207043dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.512+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.519+0000 7f207043dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:02.737 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.671+0000 7f207043dec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T20:48:02.737 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.671+0000 7f207043dec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T20:48:02.737 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.671+0000 7f207043dec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T20:48:02.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.671+0000 7f207043dec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T20:48:02.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.671+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.683+0000 7f207043dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:02.749 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.683+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.692+0000 7f207043dec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:48:02.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.692+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.700+0000 7f207043dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:02.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.700+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.709+0000 7f207043dec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:48:02.775 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.709+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.718+0000 7f207043dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.718+0000 7f207043dec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T20:48:02.784 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.718+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.723+0000 7f207043dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:02.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.723+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.800 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.735+0000 7f207043dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:02.800 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.735+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.748+0000 7f207043dec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:48:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.748+0000 7f207043dec0 20 zonegroup default 2019-12-09T20:48:02.814 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.748+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.753+0000 7f207043dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.753+0000 7f207043dec0 10 Cannot find current period zone using local zone 2019-12-09T20:48:02.819 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.753+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.826 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.761+0000 7f207043dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:02.827 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.761+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.837 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.772+0000 7f207043dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:02.838 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.772+0000 7f207043dec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.781+0000 7f207043dec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:48:02.847 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.781+0000 7f207043dec0 20 zone default found 2019-12-09T20:48:02.848 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.782+0000 7f207043dec0 20 started sync module instance, tier type = 2019-12-09T20:48:02.848 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:02.782+0000 7f207043dec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T20:48:03.220 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.154+0000 7f207043dec0 20 add_watcher() i=0 2019-12-09T20:48:03.220 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.154+0000 7f207043dec0 20 add_watcher() i=1 2019-12-09T20:48:03.220 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.154+0000 7f207043dec0 20 add_watcher() i=2 2019-12-09T20:48:03.220 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.154+0000 7f207043dec0 20 add_watcher() i=3 2019-12-09T20:48:03.220 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.154+0000 7f207043dec0 20 add_watcher() i=4 2019-12-09T20:48:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.154+0000 7f207043dec0 20 add_watcher() i=5 2019-12-09T20:48:03.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.180+0000 7f207043dec0 20 add_watcher() i=6 2019-12-09T20:48:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.208+0000 7f207043dec0 20 add_watcher() i=7 2019-12-09T20:48:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.208+0000 7f207043dec0 2 all 8 watchers are set, enabling cache 2019-12-09T20:48:03.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.209+0000 7f202dfdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:48:03.280 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.215+0000 7f205de68700 0 -- 172.21.15.67:0/1731926134 >> v2:172.21.15.81:3300/0 conn(0x55a2003987f0 msgr2=0x55a200398c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T20:48:03.483 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.418+0000 7f205de68700 0 --2- 172.21.15.67:0/1731926134 >> v2:172.21.15.81:3300/0 conn(0x55a2003987f0 0x55a200398c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-09T20:48:03.530 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:03.465+0000 7f205de68700 0 -- 172.21.15.67:0/7733038 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55a20039ede0 msgr2=0x55a20039f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:48:05.296 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.230+0000 7f1f9ffff700 20 reqs_thread_entry: start 2019-12-09T20:48:05.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.230+0000 7f1f9dffb700 20 reqs_thread_entry: start 2019-12-09T20:48:05.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.230+0000 7f207043dec0 20 init_complete bucket index max shards: 0 2019-12-09T20:48:05.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.231+0000 7f1f8bfff700 20 reqs_thread_entry: start 2019-12-09T20:48:05.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.231+0000 7f207043dec0 20 get_system_obj_state: rctx=0x7ffc2906cc18 obj=default.rgw.meta:users.uid:tester.client.0 state=0x55a20041bf10 s->prefetch_data=0 2019-12-09T20:48:05.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.231+0000 7f207043dec0 10 cache get: name=default.rgw.meta+users.uid+tester.client.0 : miss 2019-12-09T20:48:05.311 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.245+0000 7f207043dec0 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x0 2019-12-09T20:48:05.311 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.245+0000 7f207043dec0 10 adding default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2019-12-09T20:48:05.311 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.245+0000 7f207043dec0 20 get_system_obj_state: rctx=0x7ffc2906cc28 obj=default.rgw.meta:users.email:tester.client.0_test@test.test state=0x55a2003776b0 s->prefetch_data=0 2019-12-09T20:48:05.311 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.245+0000 7f207043dec0 10 cache get: name=default.rgw.meta+users.email+tester.client.0_test@test.test : miss 2019-12-09T20:48:05.317 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.252+0000 7f207043dec0 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x0 2019-12-09T20:48:05.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.252+0000 7f207043dec0 10 adding default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2019-12-09T20:48:05.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.252+0000 7f207043dec0 20 get_system_obj_state: rctx=0x7ffc2906cc28 obj=default.rgw.meta:users.keys:IABVLZVYEVPWADCATUBE state=0x55a2003776b0 s->prefetch_data=0 2019-12-09T20:48:05.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.252+0000 7f207043dec0 10 cache get: name=default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE : miss 2019-12-09T20:48:05.322 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.257+0000 7f207043dec0 10 cache put: name=default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE info.flags=0x0 2019-12-09T20:48:05.322 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.257+0000 7f207043dec0 10 adding default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE to cache LRU end 2019-12-09T20:48:05.323 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.257+0000 7f207043dec0 20 get_system_obj_state: rctx=0x7ffc2906c838 obj=default.rgw.meta:users.keys:IABVLZVYEVPWADCATUBE state=0x55a2003776b0 s->prefetch_data=0 2019-12-09T20:48:05.323 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.257+0000 7f207043dec0 10 cache get: name=default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE : type miss (requested=0x6, cached=0x0) 2019-12-09T20:48:05.329 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.264+0000 7f207043dec0 10 cache put: name=default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE info.flags=0x0 2019-12-09T20:48:05.329 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.264+0000 7f207043dec0 10 moving default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE to cache LRU end 2019-12-09T20:48:05.330 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.264+0000 7f207043dec0 20 get_system_obj_state: rctx=0x7ffc2906c9b8 obj=default.rgw.meta:users.keys:IABVLZVYEVPWADCATUBE state=0x55a2003776b0 s->prefetch_data=0 2019-12-09T20:48:05.330 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.264+0000 7f207043dec0 10 cache get: name=default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE : type miss (requested=0x6, cached=0x0) 2019-12-09T20:48:05.334 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.268+0000 7f207043dec0 10 cache put: name=default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE info.flags=0x0 2019-12-09T20:48:05.334 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.268+0000 7f207043dec0 10 moving default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE to cache LRU end 2019-12-09T20:48:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.313+0000 7f207043dec0 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2019-12-09T20:48:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.313+0000 7f207043dec0 10 moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2019-12-09T20:48:05.379 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.313+0000 7f207043dec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=632 2019-12-09T20:48:05.389 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.323+0000 7f203e7fc700 10 RGWWatcher::handle_notify() notify_id 592705486853 cookie 94154276781408 notifier 4773 bl.length()=632 2019-12-09T20:48:05.389 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.323+0000 7f203e7fc700 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2019-12-09T20:48:05.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.324+0000 7f203e7fc700 10 moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2019-12-09T20:48:05.744 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.678+0000 7f207043dec0 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2019-12-09T20:48:05.744 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.678+0000 7f207043dec0 10 moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2019-12-09T20:48:05.744 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.678+0000 7f207043dec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=200 2019-12-09T20:48:05.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.694+0000 7f203e7fc700 10 RGWWatcher::handle_notify() notify_id 592705486850 cookie 94154276748992 notifier 4773 bl.length()=200 2019-12-09T20:48:05.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.694+0000 7f203e7fc700 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2019-12-09T20:48:05.761 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:05.694+0000 7f203e7fc700 10 moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2019-12-09T20:48:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.332+0000 7f207043dec0 10 cache put: name=default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE info.flags=0x7 2019-12-09T20:48:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.332+0000 7f207043dec0 10 moving default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE to cache LRU end 2019-12-09T20:48:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.332+0000 7f207043dec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=189 2019-12-09T20:48:06.409 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.343+0000 7f203e7fc700 10 RGWWatcher::handle_notify() notify_id 592705486860 cookie 94154276747120 notifier 4773 bl.length()=189 2019-12-09T20:48:06.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.344+0000 7f203e7fc700 10 cache put: name=default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE info.flags=0x7 2019-12-09T20:48:06.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.344+0000 7f203e7fc700 10 moving default.rgw.meta+users.keys+IABVLZVYEVPWADCATUBE to cache LRU end 2019-12-09T20:48:06.669 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2019-12-09T20:48:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "user_id": "tester.client.0", 2019-12-09T20:48:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "display_name": "Ms. tester.client.0", 2019-12-09T20:48:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "email": "tester.client.0_test@test.test", 2019-12-09T20:48:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "suspended": 0, 2019-12-09T20:48:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "max_buckets": 1000, 2019-12-09T20:48:06.670 INFO:teuthology.orchestra.run.smithi067.stdout: "subusers": [], 2019-12-09T20:48:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "keys": [ 2019-12-09T20:48:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-09T20:48:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "user": "tester.client.0", 2019-12-09T20:48:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "access_key": "IABVLZVYEVPWADCATUBE", 2019-12-09T20:48:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: "secret_key": "LBcA9UK2S2vidLDphkMHkTuUZwrsTLKA9xp4Zng+rvCl5S+WNZSq2g==" 2019-12-09T20:48:06.671 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-09T20:48:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-09T20:48:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "swift_keys": [], 2019-12-09T20:48:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": [], 2019-12-09T20:48:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "op_mask": "read, write, delete", 2019-12-09T20:48:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "", 2019-12-09T20:48:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "default_storage_class": "", 2019-12-09T20:48:06.672 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_tags": [], 2019-12-09T20:48:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_quota": { 2019-12-09T20:48:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T20:48:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T20:48:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T20:48:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T20:48:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T20:48:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T20:48:06.673 INFO:teuthology.orchestra.run.smithi067.stdout: "user_quota": { 2019-12-09T20:48:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T20:48:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T20:48:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T20:48:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T20:48:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T20:48:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T20:48:06.674 INFO:teuthology.orchestra.run.smithi067.stdout: "temp_url_keys": [], 2019-12-09T20:48:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "rgw", 2019-12-09T20:48:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: "mfa_ids": [] 2019-12-09T20:48:06.675 INFO:teuthology.orchestra.run.smithi067.stdout:} 2019-12-09T20:48:06.675 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:06.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.675+0000 7f207043dec0 20 remove_watcher() i=0 2019-12-09T20:48:06.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.675+0000 7f207043dec0 2 removed watcher, disabling cache 2019-12-09T20:48:06.793 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.727+0000 7f207043dec0 20 remove_watcher() i=1 2019-12-09T20:48:06.843 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.777+0000 7f207043dec0 20 remove_watcher() i=2 2019-12-09T20:48:06.910 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.843+0000 7f207043dec0 20 remove_watcher() i=3 2019-12-09T20:48:06.961 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.895+0000 7f207043dec0 20 remove_watcher() i=4 2019-12-09T20:48:07.014 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.948+0000 7f207043dec0 20 remove_watcher() i=5 2019-12-09T20:48:07.055 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:06.988+0000 7f207043dec0 20 remove_watcher() i=6 2019-12-09T20:48:07.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.026+0000 7f207043dec0 20 remove_watcher() i=7 2019-12-09T20:48:07.099 INFO:teuthology.orchestra.run.smithi067:> cat > /home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf 2019-12-09T20:48:07.165 INFO:tasks.s3tests_java:S3 Tests Java: Running tests... 2019-12-09T20:48:07.166 INFO:teuthology.orchestra.run.smithi067:> cp /home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf /home/ubuntu/cephtest/s3-tests-java/config.properties 2019-12-09T20:48:07.250 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T20:48:07.368 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.300+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.394 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.329+0000 7ff296471ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:07.395 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.329+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.338+0000 7ff296471ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.338+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.342+0000 7ff296471ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:07.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.342+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.416 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.350+0000 7ff296471ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:07.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.545+0000 7ff296471ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T20:48:07.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.545+0000 7ff296471ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T20:48:07.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.545+0000 7ff296471ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T20:48:07.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.545+0000 7ff296471ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T20:48:07.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.545+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.554+0000 7ff296471ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:07.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.554+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.562+0000 7ff296471ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:48:07.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.562+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.572+0000 7ff296471ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:07.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.572+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.589+0000 7ff296471ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:48:07.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.589+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.597+0000 7ff296471ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:07.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.597+0000 7ff296471ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T20:48:07.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.597+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.602+0000 7ff296471ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:07.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.602+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.677 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.611+0000 7ff296471ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:07.678 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.612+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.685 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.620+0000 7ff296471ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:48:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.620+0000 7ff296471ec0 20 zonegroup default 2019-12-09T20:48:07.686 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.620+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.625+0000 7ff296471ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:07.691 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.625+0000 7ff296471ec0 10 Cannot find current period zone using local zone 2019-12-09T20:48:07.691 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.625+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.630+0000 7ff296471ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:07.696 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.630+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.704 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.639+0000 7ff296471ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:07.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.639+0000 7ff296471ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:07.713 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.647+0000 7ff296471ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:48:07.713 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.647+0000 7ff296471ec0 20 zone default found 2019-12-09T20:48:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.647+0000 7ff296471ec0 20 started sync module instance, tier type = 2019-12-09T20:48:07.714 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:07.647+0000 7ff296471ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T20:48:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.036+0000 7ff296471ec0 20 add_watcher() i=0 2019-12-09T20:48:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.036+0000 7ff296471ec0 20 add_watcher() i=1 2019-12-09T20:48:08.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.036+0000 7ff296471ec0 20 add_watcher() i=2 2019-12-09T20:48:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.036+0000 7ff296471ec0 20 add_watcher() i=3 2019-12-09T20:48:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.036+0000 7ff296471ec0 20 add_watcher() i=4 2019-12-09T20:48:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.036+0000 7ff296471ec0 20 add_watcher() i=5 2019-12-09T20:48:08.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.036+0000 7ff296471ec0 20 add_watcher() i=6 2019-12-09T20:48:08.136 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.070+0000 7ff296471ec0 20 add_watcher() i=7 2019-12-09T20:48:08.136 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.070+0000 7ff296471ec0 2 all 8 watchers are set, enabling cache 2019-12-09T20:48:08.136 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.070+0000 7ff253fdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:48:08.140 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.074+0000 7ff28369b700 0 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x56511690f490 msgr2=0x5651169290a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T20:48:08.249 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:08.183+0000 7ff28369b700 0 -- 172.21.15.67:0/1116277785 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x565116939410 msgr2=0x56511693b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:48:09.920 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:09.852+0000 7ff1c27fc700 20 reqs_thread_entry: start 2019-12-09T20:48:09.920 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:09.853+0000 7ff1bbfff700 20 reqs_thread_entry: start 2019-12-09T20:48:09.920 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:09.853+0000 7ff296471ec0 20 init_complete bucket index max shards: 0 2019-12-09T20:48:09.921 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:09.853+0000 7ff1ba7fc700 20 reqs_thread_entry: start 2019-12-09T20:48:09.921 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:09.853+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T20:48:10.023 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:09.956+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:10.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:09.985+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:10.098 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.030+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:10.098 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.030+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.225', time=2019-12-09T22:47:40.454404+0000, chain.objs.size()=5 2019-12-09T20:48:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.030+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.1 2019-12-09T20:48:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.030+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.1_1 2019-12-09T20:48:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.030+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.2 2019-12-09T20:48:10.099 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.031+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.2_1 2019-12-09T20:48:10.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.031+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.3 2019-12-09T20:48:10.216 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.149+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:10.364 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.297+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T20:48:10.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.365+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.393+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:10.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.438+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.438+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.235', time=2019-12-09T22:47:41.584876+0000, chain.objs.size()=5 2019-12-09T20:48:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.438+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.1 2019-12-09T20:48:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.438+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.1_1 2019-12-09T20:48:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.438+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.2 2019-12-09T20:48:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.438+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.2_1 2019-12-09T20:48:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.438+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.3 2019-12-09T20:48:10.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.536+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:10.807 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.740+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T20:48:10.895 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.828+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:10.921 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.854+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T20:48:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.245', time=2019-12-09T22:47:42.465464+0000, chain.objs.size()=3 2019-12-09T20:48:10.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-STANDARD.2~9btFkCibgnMZMG1P05w66RqPb2lp6to.1 2019-12-09T20:48:10.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-STANDARD.2~9btFkCibgnMZMG1P05w66RqPb2lp6to.2 2019-12-09T20:48:10.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-STANDARD.2~9btFkCibgnMZMG1P05w66RqPb2lp6to.3 2019-12-09T20:48:10.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.196', time=2019-12-09T22:47:43.443957+0000, chain.objs.size()=5 2019-12-09T20:48:10.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__multipart_foo-FROZEN.2~FZ4ElO2f4EgomLk6eMOMUZu0jMxe5Rn.1 2019-12-09T20:48:10.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__shadow_foo-FROZEN.2~FZ4ElO2f4EgomLk6eMOMUZu0jMxe5Rn.1_1 2019-12-09T20:48:10.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__multipart_foo-FROZEN.2~FZ4ElO2f4EgomLk6eMOMUZu0jMxe5Rn.2 2019-12-09T20:48:10.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__shadow_foo-FROZEN.2~FZ4ElO2f4EgomLk6eMOMUZu0jMxe5Rn.2_1 2019-12-09T20:48:10.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:10.896+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__multipart_foo-FROZEN.2~FZ4ElO2f4EgomLk6eMOMUZu0jMxe5Rn.3 2019-12-09T20:48:11.263 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.196+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.319+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T20:48:11.459 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.392+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:11.496 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.429+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:11.514 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.444+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:11.573 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.506+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T20:48:11.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.570+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:11.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.589+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:11.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.601+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:11.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.601+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_h5WkIRQ-9R7vhngQFiIfhUIq4369nLK', time=2019-12-09T22:47:37.139499+0000, chain.objs.size()=1 2019-12-09T20:48:11.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.601+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.NBDkROmz2qG1_C9ZG9IDSkcPWBdfdBq_1 2019-12-09T20:48:11.750 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.682+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:11.826 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.759+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T20:48:11.867 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.800+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:11.881 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.814+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:11.891 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.824+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:11.891 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.824+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_CMjLq4578WSMTM1gKETnId7816mENlS', time=2019-12-09T22:47:38.615310+0000, chain.objs.size()=1 2019-12-09T20:48:11.891 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.824+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_._QT2083pRDmbSFbk3QE-y4ujXDFuudp_0 2019-12-09T20:48:11.945 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.878+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:12.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:11.961+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T20:48:12.119 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.052+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:12.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.085+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:12.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.138+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:12.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.138+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.233', time=2019-12-09T22:47:41.190061+0000, chain.objs.size()=5 2019-12-09T20:48:12.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.138+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.1 2019-12-09T20:48:12.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.138+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.1_1 2019-12-09T20:48:12.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.138+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.2 2019-12-09T20:48:12.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.138+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.2_1 2019-12-09T20:48:12.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.138+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-LUKEWARM.2~5w_aDGrivLfZ13QBdpWIQ2bIMnTZNQU.3 2019-12-09T20:48:12.546 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.478+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:12.671 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.604+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T20:48:12.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.656+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:12.761 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.695+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:12.800 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.733+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T20:48:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.733+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_U4RkBQ3RIthuZRQrGyb_UXpHjlGoMVa', time=2019-12-09T22:47:40.944856+0000, chain.objs.size()=2 2019-12-09T20:48:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.733+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.yqPnjVwoL7EDL_c5F64mrr8WfZhO4T1_1 2019-12-09T20:48:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.734+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.yqPnjVwoL7EDL_c5F64mrr8WfZhO4T1_2 2019-12-09T20:48:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.734+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.85', time=2019-12-09T22:47:45.054547+0000, chain.objs.size()=3 2019-12-09T20:48:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.734+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6__shadow_.uevyI4BgGFBnKTN8eHvw_JQD37PzBIa_0 2019-12-09T20:48:12.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.734+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6__shadow_.uevyI4BgGFBnKTN8eHvw_JQD37PzBIa_1 2019-12-09T20:48:12.802 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:12.734+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6__shadow_.uevyI4BgGFBnKTN8eHvw_JQD37PzBIa_2 2019-12-09T20:48:13.121 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.055+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:13.233 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.167+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T20:48:13.284 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.217+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.237+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:13.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.247+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:13.353 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.287+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T20:48:13.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.342+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:13.431 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.365+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:13.442 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.376+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:13.510 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.443+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T20:48:13.583 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.517+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:13.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.545+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:13.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.561+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.561+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.108', time=2019-12-09T22:47:36.324897+0000, chain.objs.size()=2 2019-12-09T20:48:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.561+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.MpSyjM95Ynp6u8qXLQ_6e_nUxQTZI2n_0 2019-12-09T20:48:13.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.561+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.MpSyjM95Ynp6u8qXLQ_6e_nUxQTZI2n_1 2019-12-09T20:48:13.693 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.627+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:13.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.706+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T20:48:13.836 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.770+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:13.856 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.790+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.822+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.822+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.71', time=2019-12-09T22:47:34.942504+0000, chain.objs.size()=4 2019-12-09T20:48:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.822+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5__multipart_foo.2~zswEANFuAra8qIgmuXkm7cDcWgrsb-o.1 2019-12-09T20:48:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.822+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5__multipart_foo.2~zswEANFuAra8qIgmuXkm7cDcWgrsb-o.2 2019-12-09T20:48:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.822+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5__multipart_foo.2~zswEANFuAra8qIgmuXkm7cDcWgrsb-o.3 2019-12-09T20:48:13.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.822+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.5__multipart_foo.2~zswEANFuAra8qIgmuXkm7cDcWgrsb-o.4 2019-12-09T20:48:14.036 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:13.970+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:14.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.085+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T20:48:14.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.132+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:14.217 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.151+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:14.228 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.161+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:14.262 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.195+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T20:48:14.332 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.265+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:14.352 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.285+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:14.364 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.297+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:14.364 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.297+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_pyIq-WPxvn7M17pjhpviAE_zgoqzGME', time=2019-12-09T22:47:40.686461+0000, chain.objs.size()=3 2019-12-09T20:48:14.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.297+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.3H9ou-aii8aQI7RBxo45PVnpFWVCfqY_0 2019-12-09T20:48:14.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.297+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.3H9ou-aii8aQI7RBxo45PVnpFWVCfqY_1 2019-12-09T20:48:14.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.297+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.3H9ou-aii8aQI7RBxo45PVnpFWVCfqY_2 2019-12-09T20:48:14.575 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.508+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:14.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.674+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T20:48:14.844 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.778+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:14.886 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.819+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:14.909 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.842+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:14.964 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.897+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T20:48:15.007 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.941+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:15.022 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.956+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:15.033 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.967+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:15.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.967+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_4yuOBfWe49vaCrJ4XfdfHFb3mXMD7-1', time=2019-12-09T22:47:41.302039+0000, chain.objs.size()=3 2019-12-09T20:48:15.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.967+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.PGvjw5wiWQGMOjh-RrVR5XEBci4u22T_0 2019-12-09T20:48:15.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.967+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.PGvjw5wiWQGMOjh-RrVR5XEBci4u22T_1 2019-12-09T20:48:15.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:14.967+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.PGvjw5wiWQGMOjh-RrVR5XEBci4u22T_2 2019-12-09T20:48:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.247+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:15.494 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.428+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T20:48:15.572 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.506+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:15.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.536+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:15.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T20:48:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.114', time=2019-12-09T22:47:37.809107+0000, chain.objs.size()=2 2019-12-09T20:48:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.qq2UXZLg9aH1KFr76cr2xHpoLHPNqNI_0 2019-12-09T20:48:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.qq2UXZLg9aH1KFr76cr2xHpoLHPNqNI_1 2019-12-09T20:48:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.224', time=2019-12-09T22:47:40.338779+0000, chain.objs.size()=5 2019-12-09T20:48:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.1 2019-12-09T20:48:15.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.1_1 2019-12-09T20:48:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.2 2019-12-09T20:48:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.2_1 2019-12-09T20:48:15.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.585+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-FROZEN.2~GUJo77jCLoResdNoFMhJPRC9A_KkwEj.3 2019-12-09T20:48:15.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:15.813+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:16.067 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.000+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T20:48:16.140 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.074+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:16.168 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.101+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:16.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2019-12-09T20:48:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.20', time=2019-12-09T22:47:30.992957+0000, chain.objs.size()=4 2019-12-09T20:48:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2__multipart_foo.2~kKfr_Elfvx_JSuEmbdeRrxtOSIFSfmx.1 2019-12-09T20:48:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2__multipart_foo.2~kKfr_Elfvx_JSuEmbdeRrxtOSIFSfmx.2 2019-12-09T20:48:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2__multipart_foo.2~kKfr_Elfvx_JSuEmbdeRrxtOSIFSfmx.3 2019-12-09T20:48:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.2__multipart_foo.2~kKfr_Elfvx_JSuEmbdeRrxtOSIFSfmx.4 2019-12-09T20:48:16.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_P5oX4wEsw7pJYePy0x5Bv7I7eef9AIh', time=2019-12-09T22:47:37.552099+0000, chain.objs.size()=2 2019-12-09T20:48:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.tKbCCrvPCKt_LM7EAnrlxzHJtuepn3J_0 2019-12-09T20:48:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.tKbCCrvPCKt_LM7EAnrlxzHJtuepn3J_1 2019-12-09T20:48:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_zvpJd12a_GRr_JS8PwueVMRuTpQo29Q', time=2019-12-09T22:47:38.344485+0000, chain.objs.size()=1 2019-12-09T20:48:16.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.132+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.0N9lNub46bD0EAId_mnkj6Kq2WHPp_w_0 2019-12-09T20:48:16.449 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.383+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:16.547 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.480+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T20:48:16.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.527+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:16.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.548+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:16.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.563+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:16.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.609+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T20:48:16.717 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.650+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:16.732 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.666+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:16.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.679+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:16.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.679+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.40', time=2019-12-09T22:47:32.204983+0000, chain.objs.size()=4 2019-12-09T20:48:16.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.679+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3__multipart_foo.2~CSmch95G9HV9QwxufZwXLhWo-j8X-ZM.1 2019-12-09T20:48:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.679+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3__multipart_foo.2~CSmch95G9HV9QwxufZwXLhWo-j8X-ZM.2 2019-12-09T20:48:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.679+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3__multipart_foo.2~CSmch95G9HV9QwxufZwXLhWo-j8X-ZM.3 2019-12-09T20:48:16.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.679+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.3__multipart_foo.2~CSmch95G9HV9QwxufZwXLhWo-j8X-ZM.4 2019-12-09T20:48:16.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.896+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:17.061 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:16.994+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T20:48:17.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.036+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.051+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.062+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.062+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.210', time=2019-12-09T22:47:43.785646+0000, chain.objs.size()=3 2019-12-09T20:48:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.062+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__multipart_foo-STANDARD.2~QKpYPFi4pe445mP3tHQX4XQaqMvKIbA.1 2019-12-09T20:48:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.062+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__multipart_foo-STANDARD.2~QKpYPFi4pe445mP3tHQX4XQaqMvKIbA.2 2019-12-09T20:48:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.062+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__multipart_foo-STANDARD.2~QKpYPFi4pe445mP3tHQX4XQaqMvKIbA.3 2019-12-09T20:48:17.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.251+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:17.418 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.352+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T20:48:17.467 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.401+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:17.484 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.417+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:17.495 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.429+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T20:48:17.496 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.429+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.112', time=2019-12-09T22:47:37.383912+0000, chain.objs.size()=2 2019-12-09T20:48:17.496 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.429+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.qq2UXZLg9aH1KFr76cr2xHpoLHPNqNI_0 2019-12-09T20:48:17.496 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.429+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.qq2UXZLg9aH1KFr76cr2xHpoLHPNqNI_1 2019-12-09T20:48:17.496 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.429+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_RqnQGDlZHxcAUG2m6on92tn58bvDsLq', time=2019-12-09T22:47:42.309459+0000, chain.objs.size()=1 2019-12-09T20:48:17.496 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.429+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.nIXzkPvjKHMqbxUI_Dwk2kwgeWLNWhd_0 2019-12-09T20:48:17.720 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.654+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:17.806 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.739+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T20:48:17.892 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.825+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:17.920 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.853+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:17.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.871+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:17.994 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.927+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T20:48:18.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:17.986+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:18.070 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.003+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:18.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.020+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T20:48:18.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.020+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.242', time=2019-12-09T22:47:41.993186+0000, chain.objs.size()=3 2019-12-09T20:48:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.020+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-STANDARD.2~9btFkCibgnMZMG1P05w66RqPb2lp6to.1 2019-12-09T20:48:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.020+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-STANDARD.2~9btFkCibgnMZMG1P05w66RqPb2lp6to.2 2019-12-09T20:48:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.020+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__multipart_foo-STANDARD.2~9btFkCibgnMZMG1P05w66RqPb2lp6to.3 2019-12-09T20:48:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.020+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.84', time=2019-12-09T22:47:44.858935+0000, chain.objs.size()=3 2019-12-09T20:48:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.020+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6__shadow_.5YzEHDlnRJ9rX_GGeZGPs3fMlL7hx2q_0 2019-12-09T20:48:18.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.020+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6__shadow_.5YzEHDlnRJ9rX_GGeZGPs3fMlL7hx2q_1 2019-12-09T20:48:18.089 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.020+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.6__shadow_.5YzEHDlnRJ9rX_GGeZGPs3fMlL7hx2q_2 2019-12-09T20:48:18.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.341+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:18.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.444+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T20:48:18.560 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.493+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:18.580 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.513+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:18.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.528+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T20:48:18.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.528+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_buxWHA90m7U8nrQtO660dBMQqxiXJzJ', time=2019-12-09T22:47:36.954824+0000, chain.objs.size()=2 2019-12-09T20:48:18.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.528+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.m5gmPcfa95d1Mq2bO9X_oJEixQ-A0gK_0 2019-12-09T20:48:18.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.528+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.m5gmPcfa95d1Mq2bO9X_oJEixQ-A0gK_1 2019-12-09T20:48:18.743 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.677+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:18.887 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.820+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T20:48:18.932 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.865+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:18.950 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.883+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:18.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.892+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:19.012 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.946+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T20:48:19.057 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:18.991+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:19.074 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.007+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.020+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T20:48:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.020+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.109', time=2019-12-09T22:47:36.626742+0000, chain.objs.size()=2 2019-12-09T20:48:19.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.020+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.MpSyjM95Ynp6u8qXLQ_6e_nUxQTZI2n_0 2019-12-09T20:48:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.020+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.MpSyjM95Ynp6u8qXLQ_6e_nUxQTZI2n_1 2019-12-09T20:48:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.020+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_naSrYAjNd7Vx6e49-Xoqgwygn64i4bC', time=2019-12-09T22:47:42.105918+0000, chain.objs.size()=1 2019-12-09T20:48:19.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.020+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.jXHBpd7GBCmP3ZAxArCBjHrL4cXPSJ4_0 2019-12-09T20:48:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.213+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:19.363 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.296+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T20:48:19.420 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.353+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:19.436 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.370+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:19.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.378+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:19.496 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.429+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T20:48:19.574 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.507+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:19.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.528+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:19.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.541+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:19.688 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.621+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T20:48:19.744 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.674+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:19.765 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.699+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:19.778 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.712+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:19.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.763+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T20:48:19.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.805+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:19.891 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.824+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.838+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T20:48:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.838+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_9sTH4WyWkioClwbg4ttFOMszI8Gdmix', time=2019-12-09T22:47:37.973982+0000, chain.objs.size()=1 2019-12-09T20:48:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.838+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.7__shadow_.7SXVnU5Ydmkoxzspp6P_6cFxRgRS_kg_1 2019-12-09T20:48:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.838+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.203', time=2019-12-09T22:47:43.627240+0000, chain.objs.size()=5 2019-12-09T20:48:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.838+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__multipart_foo-LUKEWARM.2~nh8_F-QQO0Z0asnqMQWyVxdqLz3SXon.1 2019-12-09T20:48:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.838+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__shadow_foo-LUKEWARM.2~nh8_F-QQO0Z0asnqMQWyVxdqLz3SXon.1_1 2019-12-09T20:48:19.906 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.838+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__multipart_foo-LUKEWARM.2~nh8_F-QQO0Z0asnqMQWyVxdqLz3SXon.2 2019-12-09T20:48:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.838+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__shadow_foo-LUKEWARM.2~nh8_F-QQO0Z0asnqMQWyVxdqLz3SXon.2_1 2019-12-09T20:48:19.907 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:19.838+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.8__multipart_foo-LUKEWARM.2~nh8_F-QQO0Z0asnqMQWyVxdqLz3SXon.3 2019-12-09T20:48:20.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.180+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:20.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.306+0000 7ff296471ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T20:48:20.431 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.363+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:20.454 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.387+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:20.473 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.405+0000 7ff296471ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T20:48:20.473 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.405+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.55', time=2019-12-09T22:47:33.364356+0000, chain.objs.size()=4 2019-12-09T20:48:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.405+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4__multipart_foo.2~Mih_au4Pq4aXSUpj3ALzUdgqpdSOF-9.1 2019-12-09T20:48:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.405+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4__multipart_foo.2~Mih_au4Pq4aXSUpj3ALzUdgqpdSOF-9.2 2019-12-09T20:48:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.405+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4__multipart_foo.2~Mih_au4Pq4aXSUpj3ALzUdgqpdSOF-9.3 2019-12-09T20:48:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.405+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.4__multipart_foo.2~Mih_au4Pq4aXSUpj3ALzUdgqpdSOF-9.4 2019-12-09T20:48:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.405+0000 7ff296471ec0 20 garbage collection: RGWGC::process iterating over entry tag='_HF6_tqeFxfrURYsNa6XPtaoONy-VdXp', time=2019-12-09T22:47:41.693455+0000, chain.objs.size()=2 2019-12-09T20:48:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.405+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.7GfLqsWbS0I3Zv1cjlcMJOGhXpTH14u_1 2019-12-09T20:48:20.474 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.405+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.9__shadow_.7GfLqsWbS0I3Zv1cjlcMJOGhXpTH14u_2 2019-12-09T20:48:20.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.729+0000 7ff296471ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T20:48:20.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.859+0000 7ff296471ec0 20 remove_watcher() i=0 2019-12-09T20:48:20.927 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.859+0000 7ff296471ec0 2 removed watcher, disabling cache 2019-12-09T20:48:20.978 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.910+0000 7ff296471ec0 20 remove_watcher() i=1 2019-12-09T20:48:21.022 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.954+0000 7ff296471ec0 20 remove_watcher() i=2 2019-12-09T20:48:21.067 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:20.998+0000 7ff296471ec0 20 remove_watcher() i=3 2019-12-09T20:48:21.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.047+0000 7ff296471ec0 20 remove_watcher() i=4 2019-12-09T20:48:21.161 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.093+0000 7ff296471ec0 20 remove_watcher() i=5 2019-12-09T20:48:21.210 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.142+0000 7ff296471ec0 20 remove_watcher() i=6 2019-12-09T20:48:21.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.193+0000 7ff296471ec0 20 remove_watcher() i=7 2019-12-09T20:48:21.269 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T20:48:21.382 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.315+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.338+0000 7f54a8603ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:21.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.338+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.347+0000 7f54a8603ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:21.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.347+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.419 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.351+0000 7f54a8603ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:21.419 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.351+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.426 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.359+0000 7f54a8603ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.537+0000 7f54a8603ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T20:48:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.537+0000 7f54a8603ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T20:48:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.537+0000 7f54a8603ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T20:48:21.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.537+0000 7f54a8603ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T20:48:21.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.537+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.553+0000 7f54a8603ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:21.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.553+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.566+0000 7f54a8603ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:48:21.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.566+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.581+0000 7f54a8603ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:21.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.581+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.593+0000 7f54a8603ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:48:21.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.593+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.608+0000 7f54a8603ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:21.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.608+0000 7f54a8603ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T20:48:21.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.608+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.615+0000 7f54a8603ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:21.683 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.615+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.694 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.626+0000 7f54a8603ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:21.694 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.626+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.638+0000 7f54a8603ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T20:48:21.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.638+0000 7f54a8603ec0 20 zonegroup default 2019-12-09T20:48:21.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.638+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.644+0000 7f54a8603ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:21.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.644+0000 7f54a8603ec0 10 Cannot find current period zone using local zone 2019-12-09T20:48:21.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.644+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.719 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.651+0000 7f54a8603ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T20:48:21.719 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.651+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.663+0000 7f54a8603ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T20:48:21.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.663+0000 7f54a8603ec0 20 rados->read ofs=0 len=0 2019-12-09T20:48:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.677+0000 7f54a8603ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T20:48:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.678+0000 7f54a8603ec0 20 zone default found 2019-12-09T20:48:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.678+0000 7f54a8603ec0 20 started sync module instance, tier type = 2019-12-09T20:48:21.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:21.678+0000 7f54a8603ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T20:48:22.212 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.144+0000 7f54a8603ec0 20 add_watcher() i=0 2019-12-09T20:48:22.212 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.144+0000 7f54a8603ec0 20 add_watcher() i=1 2019-12-09T20:48:22.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.144+0000 7f54a8603ec0 20 add_watcher() i=2 2019-12-09T20:48:22.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.144+0000 7f54a8603ec0 20 add_watcher() i=3 2019-12-09T20:48:22.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.144+0000 7f54a8603ec0 20 add_watcher() i=4 2019-12-09T20:48:22.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.144+0000 7f54a8603ec0 20 add_watcher() i=5 2019-12-09T20:48:22.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.144+0000 7f54a8603ec0 20 add_watcher() i=6 2019-12-09T20:48:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.189+0000 7f54a8603ec0 20 add_watcher() i=7 2019-12-09T20:48:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.189+0000 7f54a8603ec0 2 all 8 watchers are set, enabling cache 2019-12-09T20:48:22.258 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.190+0000 7f54667dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T20:48:22.268 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.200+0000 7f549602e700 0 -- 172.21.15.67:0/151195547 >> v2:172.21.15.67:3300/0 conn(0x559b6213f110 msgr2=0x559b62161220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T20:48:22.296 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:22.228+0000 7f548ffff700 0 -- 172.21.15.67:0/3064276167 >> v2:172.21.15.81:3300/0 conn(0x559b6217d850 msgr2=0x559b6216de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T20:48:24.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.034+0000 7f53dcff9700 20 reqs_thread_entry: start 2019-12-09T20:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.034+0000 7f53d67fc700 20 reqs_thread_entry: start 2019-12-09T20:48:24.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.035+0000 7f54a8603ec0 20 init_complete bucket index max shards: 0 2019-12-09T20:48:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.035+0000 7f53d4ff9700 20 reqs_thread_entry: start 2019-12-09T20:48:24.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.035+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T20:48:24.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.085+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.173 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.106+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:24.184 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.117+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.218 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.151+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T20:48:24.277 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.210+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.294 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.227+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:24.305 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.238+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.343 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.276+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T20:48:24.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.339+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.426 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.359+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:24.436 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.370+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.488 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.421+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T20:48:24.551 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.484+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.501+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:24.575 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.508+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.556+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T20:48:24.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.663+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.752 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.685+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:24.761 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.694+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.821 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.753+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T20:48:24.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.821+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.908 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.841+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:24.917 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.850+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:24.957 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.890+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T20:48:24.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.932+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.016 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.949+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:25.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.958+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.056 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:24.989+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T20:48:25.120 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.053+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.074+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:25.151 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.083+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.201 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.134+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T20:48:25.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.215+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.311 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.244+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:25.326 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.259+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.371 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.304+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T20:48:25.421 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.353+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.439 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.372+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:25.448 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.381+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.484 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.417+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T20:48:25.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.501+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.535+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:25.619 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.552+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.671 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.604+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T20:48:25.724 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.657+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.680+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:25.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.693+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.728+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T20:48:25.850 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.783+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.867 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.800+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:25.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.812+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:25.911 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.844+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T20:48:25.987 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.920+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.019 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.951+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:26.035 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:25.967+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.099 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.032+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T20:48:26.187 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.119+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.224 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.157+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:26.241 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.174+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.302 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.234+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T20:48:26.397 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.328+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.378+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:26.458 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.390+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.508 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.441+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T20:48:26.733 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.666+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.759 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.691+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:26.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.705+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.814 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.746+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T20:48:26.875 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.808+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.896 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.829+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:26.909 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.842+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:26.945 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.878+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T20:48:27.004 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.937+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.020 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.953+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:26.964+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.075 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.008+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T20:48:27.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.082+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.179 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.112+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:27.195 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.128+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.197+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T20:48:27.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.277+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.298+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.307+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.439 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.372+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T20:48:27.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.439+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.535 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.468+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:27.559 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.492+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.549+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T20:48:27.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.597+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.683 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.616+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:27.717 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.650+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.755 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.688+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T20:48:27.831 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.764+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.856 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.789+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:27.869 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.801+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:27.922 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.855+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T20:48:28.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.932+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.966+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:28.051 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:27.983+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.112 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.045+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T20:48:28.167 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.099+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.185 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.118+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:28.196 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.128+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.232 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.165+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T20:48:28.301 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.234+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.322 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.255+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:28.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.269+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.400 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.333+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T20:48:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.458+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.556 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.488+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:28.564 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.497+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.534+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T20:48:28.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.579+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.600+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:28.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.609+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.638+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T20:48:28.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.698+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.730+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:28.819 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.751+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.861 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.794+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T20:48:28.912 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.844+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.928 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.860+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:28.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.869+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:28.978 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.911+0000 7f54a8603ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T20:48:29.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.956+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:29.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:28.985+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T20:48:29.071 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.004+0000 7f54a8603ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T20:48:29.140 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.073+0000 7f54a8603ec0 20 remove_watcher() i=0 2019-12-09T20:48:29.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.073+0000 7f54a8603ec0 2 removed watcher, disabling cache 2019-12-09T20:48:29.178 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.110+0000 7f54a8603ec0 20 remove_watcher() i=1 2019-12-09T20:48:29.216 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.148+0000 7f54a8603ec0 20 remove_watcher() i=2 2019-12-09T20:48:29.279 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.212+0000 7f54a8603ec0 20 remove_watcher() i=3 2019-12-09T20:48:29.337 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.269+0000 7f54a8603ec0 20 remove_watcher() i=4 2019-12-09T20:48:29.387 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.319+0000 7f54a8603ec0 20 remove_watcher() i=5 2019-12-09T20:48:29.448 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.380+0000 7f54a8603ec0 20 remove_watcher() i=6 2019-12-09T20:48:29.504 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T20:48:29.437+0000 7f54a8603ec0 20 remove_watcher() i=7 2019-12-09T20:48:29.513 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle-4.7/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test 2019-12-09T20:48:30.135 INFO:teuthology.orchestra.run.smithi067.stdout:Starting a Gradle Daemon (subsequent builds will be faster) 2019-12-09T20:48:33.335 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/spring/gradle/dependency-management-plugin/1.0.3.RELEASE/dependency-management-plugin-1.0.3.RELEASE.pom 2019-12-09T20:48:33.435 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/spring/gradle/dependency-management-plugin/1.0.3.RELEASE/dependency-management-plugin-1.0.3.RELEASE.jar 2019-12-09T20:48:34.638 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :clean UP-TO-DATE 2019-12-09T20:48:34.735 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-bom/1.11.549/aws-java-sdk-bom-1.11.549.pom 2019-12-09T20:48:34.835 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pom/1.11.549/aws-java-sdk-pom-1.11.549.pom 2019-12-09T20:48:35.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sqs/1.11.549/aws-java-sdk-sqs-1.11.549.pom 2019-12-09T20:48:35.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk/1.11.549/aws-java-sdk-1.11.549.pom 2019-12-09T20:48:35.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.549/aws-java-sdk-core-1.11.549.pom 2019-12-09T20:48:35.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-s3/1.11.549/aws-java-sdk-s3-1.11.549.pom 2019-12-09T20:48:35.037 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/testng/testng/6.1.1/testng-6.1.1.pom 2019-12-09T20:48:35.037 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.pom 2019-12-09T20:48:35.037 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-server/2.44.0/selenium-server-2.44.0.pom 2019-12-09T20:48:35.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-parent/2.44.0/selenium-parent-2.44.0.pom 2019-12-09T20:48:35.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/3/oss-parent-3.pom 2019-12-09T20:48:35.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.pom 2019-12-09T20:48:35.139 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/joda-time/joda-time/2.8.1/joda-time-2.8.1.pom 2019-12-09T20:48:35.139 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.5/httpclient-4.5.5.pom 2019-12-09T20:48:35.139 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.7.2/jackson-databind-2.6.7.2.pom 2019-12-09T20:48:35.139 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.pom 2019-12-09T20:48:35.139 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.pom 2019-12-09T20:48:35.139 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.6.2/jackson-parent-2.6.2.pom 2019-12-09T20:48:35.140 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-client/4.5.5/httpcomponents-client-4.5.5.pom 2019-12-09T20:48:35.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/oss-parent/24/oss-parent-24.pom 2019-12-09T20:48:35.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/28/commons-parent-28.pom 2019-12-09T20:48:35.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-parent/10/httpcomponents-parent-10.pom 2019-12-09T20:48:35.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom 2019-12-09T20:48:35.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/13/apache-13.pom 2019-12-09T20:48:35.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kafka/1.11.549/aws-java-sdk-kafka-1.11.549.pom 2019-12-09T20:48:35.338 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-apigatewayv2/1.11.549/aws-java-sdk-apigatewayv2-1.11.549.pom 2019-12-09T20:48:35.338 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-apigatewaymanagementapi/1.11.549/aws-java-sdk-apigatewaymanagementapi-1.11.549.pom 2019-12-09T20:48:35.338 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-textract/1.11.549/aws-java-sdk-textract-1.11.549.pom 2019-12-09T20:48:35.338 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-worklink/1.11.549/aws-java-sdk-worklink-1.11.549.pom 2019-12-09T20:48:35.339 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-docdb/1.11.549/aws-java-sdk-docdb-1.11.549.pom 2019-12-09T20:48:35.339 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-managedblockchain/1.11.549/aws-java-sdk-managedblockchain-1.11.549.pom 2019-12-09T20:48:35.339 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-licensemanager/1.11.549/aws-java-sdk-licensemanager-1.11.549.pom 2019-12-09T20:48:35.339 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-backup/1.11.549/aws-java-sdk-backup-1.11.549.pom 2019-12-09T20:48:35.339 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-fsx/1.11.549/aws-java-sdk-fsx-1.11.549.pom 2019-12-09T20:48:35.340 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediaconnect/1.11.549/aws-java-sdk-mediaconnect-1.11.549.pom 2019-12-09T20:48:35.340 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesisanalyticsv2/1.11.549/aws-java-sdk-kinesisanalyticsv2-1.11.549.pom 2019-12-09T20:48:35.340 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-comprehendmedical/1.11.549/aws-java-sdk-comprehendmedical-1.11.549.pom 2019-12-09T20:48:35.340 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-securityhub/1.11.549/aws-java-sdk-securityhub-1.11.549.pom 2019-12-09T20:48:35.341 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-amplify/1.11.549/aws-java-sdk-amplify-1.11.549.pom 2019-12-09T20:48:35.341 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-globalaccelerator/1.11.549/aws-java-sdk-globalaccelerator-1.11.549.pom 2019-12-09T20:48:35.341 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-quicksight/1.11.549/aws-java-sdk-quicksight-1.11.549.pom 2019-12-09T20:48:35.341 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-transfer/1.11.549/aws-java-sdk-transfer-1.11.549.pom 2019-12-09T20:48:35.342 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-robomaker/1.11.549/aws-java-sdk-robomaker-1.11.549.pom 2019-12-09T20:48:35.342 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-datasync/1.11.549/aws-java-sdk-datasync-1.11.549.pom 2019-12-09T20:48:35.342 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-route53resolver/1.11.549/aws-java-sdk-route53resolver-1.11.549.pom 2019-12-09T20:48:35.435 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpointsmsvoice/1.11.549/aws-java-sdk-pinpointsmsvoice-1.11.549.pom 2019-12-09T20:48:35.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-s3control/1.11.549/aws-java-sdk-s3control-1.11.549.pom 2019-12-09T20:48:35.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rdsdata/1.11.549/aws-java-sdk-rdsdata-1.11.549.pom 2019-12-09T20:48:35.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-chime/1.11.549/aws-java-sdk-chime-1.11.549.pom 2019-12-09T20:48:35.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpointemail/1.11.549/aws-java-sdk-pinpointemail-1.11.549.pom 2019-12-09T20:48:35.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ram/1.11.549/aws-java-sdk-ram-1.11.549.pom 2019-12-09T20:48:35.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-signer/1.11.549/aws-java-sdk-signer-1.11.549.pom 2019-12-09T20:48:35.440 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-eks/1.11.549/aws-java-sdk-eks-1.11.549.pom 2019-12-09T20:48:35.440 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dlm/1.11.549/aws-java-sdk-dlm-1.11.549.pom 2019-12-09T20:48:35.440 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-macie/1.11.549/aws-java-sdk-macie-1.11.549.pom 2019-12-09T20:48:35.440 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pi/1.11.549/aws-java-sdk-pi-1.11.549.pom 2019-12-09T20:48:35.440 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-neptune/1.11.549/aws-java-sdk-neptune-1.11.549.pom 2019-12-09T20:48:35.440 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediatailor/1.11.549/aws-java-sdk-mediatailor-1.11.549.pom 2019-12-09T20:48:35.441 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot1clickdevices/1.11.549/aws-java-sdk-iot1clickdevices-1.11.549.pom 2019-12-09T20:48:35.441 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot1clickprojects/1.11.549/aws-java-sdk-iot1clickprojects-1.11.549.pom 2019-12-09T20:48:35.441 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iotanalytics/1.11.549/aws-java-sdk-iotanalytics-1.11.549.pom 2019-12-09T20:48:35.441 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-acmpca/1.11.549/aws-java-sdk-acmpca-1.11.549.pom 2019-12-09T20:48:35.441 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-secretsmanager/1.11.549/aws-java-sdk-secretsmanager-1.11.549.pom 2019-12-09T20:48:35.441 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-fms/1.11.549/aws-java-sdk-fms-1.11.549.pom 2019-12-09T20:48:35.441 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-connect/1.11.549/aws-java-sdk-connect-1.11.549.pom 2019-12-09T20:48:35.442 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-transcribe/1.11.549/aws-java-sdk-transcribe-1.11.549.pom 2019-12-09T20:48:35.442 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-autoscalingplans/1.11.549/aws-java-sdk-autoscalingplans-1.11.549.pom 2019-12-09T20:48:35.442 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servicediscovery/1.11.549/aws-java-sdk-servicediscovery-1.11.549.pom 2019-12-09T20:48:35.442 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloud9/1.11.549/aws-java-sdk-cloud9-1.11.549.pom 2019-12-09T20:48:35.442 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workmail/1.11.549/aws-java-sdk-workmail-1.11.549.pom 2019-12-09T20:48:35.443 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-serverlessapplicationrepository/1.11.549/aws-java-sdk-serverlessapplicationrepository-1.11.549.pom 2019-12-09T20:48:35.443 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-alexaforbusiness/1.11.549/aws-java-sdk-alexaforbusiness-1.11.549.pom 2019-12-09T20:48:35.443 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-comprehend/1.11.549/aws-java-sdk-comprehend-1.11.549.pom 2019-12-09T20:48:35.443 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-translate/1.11.549/aws-java-sdk-translate-1.11.549.pom 2019-12-09T20:48:35.443 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-resourcegroups/1.11.549/aws-java-sdk-resourcegroups-1.11.549.pom 2019-12-09T20:48:35.444 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iotjobsdataplane/1.11.549/aws-java-sdk-iotjobsdataplane-1.11.549.pom 2019-12-09T20:48:35.444 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sagemakerruntime/1.11.549/aws-java-sdk-sagemakerruntime-1.11.549.pom 2019-12-09T20:48:35.444 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sagemaker/1.11.549/aws-java-sdk-sagemaker-1.11.549.pom 2019-12-09T20:48:35.444 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesisvideo/1.11.549/aws-java-sdk-kinesisvideo-1.11.549.pom 2019-12-09T20:48:35.445 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appsync/1.11.549/aws-java-sdk-appsync-1.11.549.pom 2019-12-09T20:48:35.445 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mq/1.11.549/aws-java-sdk-mq-1.11.549.pom 2019-12-09T20:48:35.535 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-guardduty/1.11.549/aws-java-sdk-guardduty-1.11.549.pom 2019-12-09T20:48:35.536 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediastore/1.11.549/aws-java-sdk-mediastore-1.11.549.pom 2019-12-09T20:48:35.536 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediapackage/1.11.549/aws-java-sdk-mediapackage-1.11.549.pom 2019-12-09T20:48:35.536 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediastoredata/1.11.549/aws-java-sdk-mediastoredata-1.11.549.pom 2019-12-09T20:48:35.536 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-medialive/1.11.549/aws-java-sdk-medialive-1.11.549.pom 2019-12-09T20:48:35.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediaconvert/1.11.549/aws-java-sdk-mediaconvert-1.11.549.pom 2019-12-09T20:48:35.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-costexplorer/1.11.549/aws-java-sdk-costexplorer-1.11.549.pom 2019-12-09T20:48:35.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pricing/1.11.549/aws-java-sdk-pricing-1.11.549.pom 2019-12-09T20:48:35.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudhsmv2/1.11.549/aws-java-sdk-cloudhsmv2-1.11.549.pom 2019-12-09T20:48:35.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mobile/1.11.549/aws-java-sdk-mobile-1.11.549.pom 2019-12-09T20:48:35.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-migrationhub/1.11.549/aws-java-sdk-migrationhub-1.11.549.pom 2019-12-09T20:48:35.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-glue/1.11.549/aws-java-sdk-glue-1.11.549.pom 2019-12-09T20:48:35.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-athena/1.11.549/aws-java-sdk-athena-1.11.549.pom 2019-12-09T20:48:35.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-greengrass/1.11.549/aws-java-sdk-greengrass-1.11.549.pom 2019-12-09T20:48:35.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dax/1.11.549/aws-java-sdk-dax-1.11.549.pom 2019-12-09T20:48:35.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codestar/1.11.549/aws-java-sdk-codestar-1.11.549.pom 2019-12-09T20:48:35.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lexmodelbuilding/1.11.549/aws-java-sdk-lexmodelbuilding-1.11.549.pom 2019-12-09T20:48:35.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplaceentitlement/1.11.549/aws-java-sdk-marketplaceentitlement-1.11.549.pom 2019-12-09T20:48:35.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-resourcegroupstaggingapi/1.11.549/aws-java-sdk-resourcegroupstaggingapi-1.11.549.pom 2019-12-09T20:48:35.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpoint/1.11.549/aws-java-sdk-pinpoint-1.11.549.pom 2019-12-09T20:48:35.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-xray/1.11.549/aws-java-sdk-xray-1.11.549.pom 2019-12-09T20:48:35.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-support/1.11.549/aws-java-sdk-support-1.11.549.pom 2019-12-09T20:48:35.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-simpleworkflow/1.11.549/aws-java-sdk-simpleworkflow-1.11.549.pom 2019-12-09T20:48:35.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-simpledb/1.11.549/aws-java-sdk-simpledb-1.11.549.pom 2019-12-09T20:48:35.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servicecatalog/1.11.549/aws-java-sdk-servicecatalog-1.11.549.pom 2019-12-09T20:48:35.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-storagegateway/1.11.549/aws-java-sdk-storagegateway-1.11.549.pom 2019-12-09T20:48:35.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-opsworkscm/1.11.549/aws-java-sdk-opsworkscm-1.11.549.pom 2019-12-09T20:48:35.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servermigration/1.11.549/aws-java-sdk-servermigration-1.11.549.pom 2019-12-09T20:48:35.637 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-route53/1.11.549/aws-java-sdk-route53-1.11.549.pom 2019-12-09T20:48:35.637 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-importexport/1.11.549/aws-java-sdk-importexport-1.11.549.pom 2019-12-09T20:48:35.638 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rds/1.11.549/aws-java-sdk-rds-1.11.549.pom 2019-12-09T20:48:35.638 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-redshift/1.11.549/aws-java-sdk-redshift-1.11.549.pom 2019-12-09T20:48:35.638 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sts/1.11.549/aws-java-sdk-sts-1.11.549.pom 2019-12-09T20:48:35.638 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-glacier/1.11.549/aws-java-sdk-glacier-1.11.549.pom 2019-12-09T20:48:35.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticbeanstalk/1.11.549/aws-java-sdk-elasticbeanstalk-1.11.549.pom 2019-12-09T20:48:35.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-datapipeline/1.11.549/aws-java-sdk-datapipeline-1.11.549.pom 2019-12-09T20:48:35.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticloadbalancing/1.11.549/aws-java-sdk-elasticloadbalancing-1.11.549.pom 2019-12-09T20:48:35.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iam/1.11.549/aws-java-sdk-iam-1.11.549.pom 2019-12-09T20:48:35.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-emr/1.11.549/aws-java-sdk-emr-1.11.549.pom 2019-12-09T20:48:35.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elastictranscoder/1.11.549/aws-java-sdk-elastictranscoder-1.11.549.pom 2019-12-09T20:48:35.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticloadbalancingv2/1.11.549/aws-java-sdk-elasticloadbalancingv2-1.11.549.pom 2019-12-09T20:48:35.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticache/1.11.549/aws-java-sdk-elasticache-1.11.549.pom 2019-12-09T20:48:35.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dynamodb/1.11.549/aws-java-sdk-dynamodb-1.11.549.pom 2019-12-09T20:48:35.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sns/1.11.549/aws-java-sdk-sns-1.11.549.pom 2019-12-09T20:48:35.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ec2/1.11.549/aws-java-sdk-ec2-1.11.549.pom 2019-12-09T20:48:35.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-budgets/1.11.549/aws-java-sdk-budgets-1.11.549.pom 2019-12-09T20:48:35.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudwatch/1.11.549/aws-java-sdk-cloudwatch-1.11.549.pom 2019-12-09T20:48:35.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-logs/1.11.549/aws-java-sdk-logs-1.11.549.pom 2019-12-09T20:48:35.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-events/1.11.549/aws-java-sdk-events-1.11.549.pom 2019-12-09T20:48:35.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitosync/1.11.549/aws-java-sdk-cognitosync-1.11.549.pom 2019-12-09T20:48:35.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudtrail/1.11.549/aws-java-sdk-cloudtrail-1.11.549.pom 2019-12-09T20:48:35.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitoidentity/1.11.549/aws-java-sdk-cognitoidentity-1.11.549.pom 2019-12-09T20:48:35.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-directconnect/1.11.549/aws-java-sdk-directconnect-1.11.549.pom 2019-12-09T20:48:35.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudformation/1.11.549/aws-java-sdk-cloudformation-1.11.549.pom 2019-12-09T20:48:35.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudfront/1.11.549/aws-java-sdk-cloudfront-1.11.549.pom 2019-12-09T20:48:35.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesis/1.11.549/aws-java-sdk-kinesis-1.11.549.pom 2019-12-09T20:48:35.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-clouddirectory/1.11.549/aws-java-sdk-clouddirectory-1.11.549.pom 2019-12-09T20:48:35.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-opsworks/1.11.549/aws-java-sdk-opsworks-1.11.549.pom 2019-12-09T20:48:35.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-autoscaling/1.11.549/aws-java-sdk-autoscaling-1.11.549.pom 2019-12-09T20:48:35.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ses/1.11.549/aws-java-sdk-ses-1.11.549.pom 2019-12-09T20:48:35.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudwatchmetrics/1.11.549/aws-java-sdk-cloudwatchmetrics-1.11.549.pom 2019-12-09T20:48:35.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudsearch/1.11.549/aws-java-sdk-cloudsearch-1.11.549.pom 2019-12-09T20:48:35.643 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codedeploy/1.11.549/aws-java-sdk-codedeploy-1.11.549.pom 2019-12-09T20:48:35.643 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codepipeline/1.11.549/aws-java-sdk-codepipeline-1.11.549.pom 2019-12-09T20:48:35.643 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kms/1.11.549/aws-java-sdk-kms-1.11.549.pom 2019-12-09T20:48:35.643 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lambda/1.11.549/aws-java-sdk-lambda-1.11.549.pom 2019-12-09T20:48:35.643 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudhsm/1.11.549/aws-java-sdk-cloudhsm-1.11.549.pom 2019-12-09T20:48:35.643 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ecr/1.11.549/aws-java-sdk-ecr-1.11.549.pom 2019-12-09T20:48:35.644 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-config/1.11.549/aws-java-sdk-config-1.11.549.pom 2019-12-09T20:48:35.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ecs/1.11.549/aws-java-sdk-ecs-1.11.549.pom 2019-12-09T20:48:35.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ssm/1.11.549/aws-java-sdk-ssm-1.11.549.pom 2019-12-09T20:48:35.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workspaces/1.11.549/aws-java-sdk-workspaces-1.11.549.pom 2019-12-09T20:48:35.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-machinelearning/1.11.549/aws-java-sdk-machinelearning-1.11.549.pom 2019-12-09T20:48:35.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-efs/1.11.549/aws-java-sdk-efs-1.11.549.pom 2019-12-09T20:48:35.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codecommit/1.11.549/aws-java-sdk-codecommit-1.11.549.pom 2019-12-09T20:48:35.747 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-directory/1.11.549/aws-java-sdk-directory-1.11.549.pom 2019-12-09T20:48:35.747 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-devicefarm/1.11.549/aws-java-sdk-devicefarm-1.11.549.pom 2019-12-09T20:48:35.748 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticsearch/1.11.549/aws-java-sdk-elasticsearch-1.11.549.pom 2019-12-09T20:48:35.748 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplacecommerceanalytics/1.11.549/aws-java-sdk-marketplacecommerceanalytics-1.11.549.pom 2019-12-09T20:48:35.748 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-waf/1.11.549/aws-java-sdk-waf-1.11.549.pom 2019-12-09T20:48:35.748 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-inspector/1.11.549/aws-java-sdk-inspector-1.11.549.pom 2019-12-09T20:48:35.749 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot/1.11.549/aws-java-sdk-iot-1.11.549.pom 2019-12-09T20:48:35.749 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-gamelift/1.11.549/aws-java-sdk-gamelift-1.11.549.pom 2019-12-09T20:48:35.749 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dms/1.11.549/aws-java-sdk-dms-1.11.549.pom 2019-12-09T20:48:35.749 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplacemeteringservice/1.11.549/aws-java-sdk-marketplacemeteringservice-1.11.549.pom 2019-12-09T20:48:35.749 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitoidp/1.11.549/aws-java-sdk-cognitoidp-1.11.549.pom 2019-12-09T20:48:35.749 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-discovery/1.11.549/aws-java-sdk-discovery-1.11.549.pom 2019-12-09T20:48:35.749 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-api-gateway/1.11.549/aws-java-sdk-api-gateway-1.11.549.pom 2019-12-09T20:48:35.750 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-snowball/1.11.549/aws-java-sdk-snowball-1.11.549.pom 2019-12-09T20:48:35.750 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-applicationautoscaling/1.11.549/aws-java-sdk-applicationautoscaling-1.11.549.pom 2019-12-09T20:48:35.750 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-acm/1.11.549/aws-java-sdk-acm-1.11.549.pom 2019-12-09T20:48:35.750 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-polly/1.11.549/aws-java-sdk-polly-1.11.549.pom 2019-12-09T20:48:35.750 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rekognition/1.11.549/aws-java-sdk-rekognition-1.11.549.pom 2019-12-09T20:48:35.750 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-health/1.11.549/aws-java-sdk-health-1.11.549.pom 2019-12-09T20:48:35.750 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-costandusagereport/1.11.549/aws-java-sdk-costandusagereport-1.11.549.pom 2019-12-09T20:48:35.751 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-stepfunctions/1.11.549/aws-java-sdk-stepfunctions-1.11.549.pom 2019-12-09T20:48:35.751 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codebuild/1.11.549/aws-java-sdk-codebuild-1.11.549.pom 2019-12-09T20:48:35.751 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appstream/1.11.549/aws-java-sdk-appstream-1.11.549.pom 2019-12-09T20:48:35.751 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lightsail/1.11.549/aws-java-sdk-lightsail-1.11.549.pom 2019-12-09T20:48:35.751 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-shield/1.11.549/aws-java-sdk-shield-1.11.549.pom 2019-12-09T20:48:35.751 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-batch/1.11.549/aws-java-sdk-batch-1.11.549.pom 2019-12-09T20:48:35.751 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mechanicalturkrequester/1.11.549/aws-java-sdk-mechanicalturkrequester-1.11.549.pom 2019-12-09T20:48:35.752 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workdocs/1.11.549/aws-java-sdk-workdocs-1.11.549.pom 2019-12-09T20:48:35.752 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lex/1.11.549/aws-java-sdk-lex-1.11.549.pom 2019-12-09T20:48:35.752 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-organizations/1.11.549/aws-java-sdk-organizations-1.11.549.pom 2019-12-09T20:48:35.752 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appmesh/1.11.549/aws-java-sdk-appmesh-1.11.549.pom 2019-12-09T20:48:35.752 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-models/1.11.549/aws-java-sdk-models-1.11.549.pom 2019-12-09T20:48:35.752 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-swf-libraries/1.11.22/aws-java-sdk-swf-libraries-1.11.22.pom 2019-12-09T20:48:35.863 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pom/1.11.22/aws-java-sdk-pom-1.11.22.pom 2019-12-09T20:48:35.865 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/jmespath-java/1.11.549/jmespath-java-1.11.549.pom 2019-12-09T20:48:35.865 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcprov-jdk15on/1.48/bcprov-jdk15on-1.48.pom 2019-12-09T20:48:35.865 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-java/2.44.0/selenium-java-2.44.0.pom 2019-12-09T20:48:35.866 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.8/snakeyaml-1.8.pom 2019-12-09T20:48:35.866 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.pom 2019-12-09T20:48:35.866 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/jetty-rc-repacked/5/jetty-rc-repacked-5.pom 2019-12-09T20:48:35.868 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/jetty-repacked/7.6.1/jetty-repacked-7.6.1.pom 2019-12-09T20:48:35.868 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.48/bcpkix-jdk15on-1.48.pom 2019-12-09T20:48:35.868 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/mx4j/mx4j-tools/3.0.1/mx4j-tools-3.0.1.pom 2019-12-09T20:48:35.869 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.9/commons-codec-1.9.pom 2019-12-09T20:48:35.869 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/mortbay/jetty/servlet-api-2.5/6.1.9/servlet-api-2.5-6.1.9.pom 2019-12-09T20:48:35.869 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/32/commons-parent-32.pom 2019-12-09T20:48:35.871 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/mortbay/jetty/project/6.1.9/project-6.1.9.pom 2019-12-09T20:48:35.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/junit/junit/3.8.1/junit-3.8.1.pom 2019-12-09T20:48:35.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/beust/jcommander/1.12/jcommander-1.12.pom 2019-12-09T20:48:35.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/beanshell/bsh/2.0b4/bsh-2.0b4.pom 2019-12-09T20:48:35.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/beanshell/beanshell/2.0b4/beanshell-2.0b4.pom 2019-12-09T20:48:35.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.pom 2019-12-09T20:48:35.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/34/commons-parent-34.pom 2019-12-09T20:48:35.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.pom 2019-12-09T20:48:35.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/35/commons-parent-35.pom 2019-12-09T20:48:35.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/15/apache-15.pom 2019-12-09T20:48:36.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.pom 2019-12-09T20:48:36.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-core/4.4.9/httpcomponents-core-4.4.9.pom 2019-12-09T20:48:36.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-parent/9/httpcomponents-parent-9.pom 2019-12-09T20:48:36.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.pom 2019-12-09T20:48:36.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.pom 2019-12-09T20:48:36.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/jackson-parent/2.6.1/jackson-parent-2.6.1.pom 2019-12-09T20:48:36.037 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/oss-parent/23/oss-parent-23.pom 2019-12-09T20:48:36.137 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-codec-http/4.1.17.Final/netty-codec-http-4.1.17.Final.pom 2019-12-09T20:48:36.137 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-handler/4.1.17.Final/netty-handler-4.1.17.Final.pom 2019-12-09T20:48:36.137 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-parent/4.1.17.Final/netty-parent-4.1.17.Final.pom 2019-12-09T20:48:36.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/9/oss-parent-9.pom 2019-12-09T20:48:36.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-chrome-driver/2.44.0/selenium-chrome-driver-2.44.0.pom 2019-12-09T20:48:36.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-support/2.44.0/selenium-support-2.44.0.pom 2019-12-09T20:48:36.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-safari-driver/2.44.0/selenium-safari-driver-2.44.0.pom 2019-12-09T20:48:36.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-firefox-driver/2.44.0/selenium-firefox-driver-2.44.0.pom 2019-12-09T20:48:36.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-htmlunit-driver/2.44.0/selenium-htmlunit-driver-2.44.0.pom 2019-12-09T20:48:36.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-ie-driver/2.44.0/selenium-ie-driver-2.44.0.pom 2019-12-09T20:48:36.139 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/webbitserver/webbit/0.4.14/webbit-0.4.14.pom 2019-12-09T20:48:36.139 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/6/oss-parent-6.pom 2019-12-09T20:48:36.139 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-codec/4.1.17.Final/netty-codec-4.1.17.Final.pom 2019-12-09T20:48:36.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-transport/4.1.17.Final/netty-transport-4.1.17.Final.pom 2019-12-09T20:48:36.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-buffer/4.1.17.Final/netty-buffer-4.1.17.Final.pom 2019-12-09T20:48:36.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-remote-driver/2.44.0/selenium-remote-driver-2.44.0.pom 2019-12-09T20:48:36.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/htmlunit/htmlunit/2.15/htmlunit-2.15.pom 2019-12-09T20:48:36.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-io/commons-io/2.4/commons-io-2.4.pom 2019-12-09T20:48:36.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-exec/1.1/commons-exec-1.1.pom 2019-12-09T20:48:36.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/17/commons-parent-17.pom 2019-12-09T20:48:36.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/25/commons-parent-25.pom 2019-12-09T20:48:36.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/7/apache-7.pom 2019-12-09T20:48:36.238 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/9/apache-9.pom 2019-12-09T20:48:36.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/java/dev/jna/jna/3.4.0/jna-3.4.0.pom 2019-12-09T20:48:36.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/java/dev/jna/platform/3.4.0/platform-3.4.0.pom 2019-12-09T20:48:36.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty/3.5.2.Final/netty-3.5.2.Final.pom 2019-12-09T20:48:36.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/sonatype/oss/oss-parent/7/oss-parent-7.pom 2019-12-09T20:48:36.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-common/4.1.17.Final/netty-common-4.1.17.Final.pom 2019-12-09T20:48:36.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-resolver/4.1.17.Final/netty-resolver-4.1.17.Final.pom 2019-12-09T20:48:36.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/google/guava/guava/18.0/guava-18.0.pom 2019-12-09T20:48:36.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-api/2.44.0/selenium-api-2.44.0.pom 2019-12-09T20:48:36.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/cglib/cglib-nodep/2.1_3/cglib-nodep-2.1_3.pom 2019-12-09T20:48:36.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.3/gson-2.3.pom 2019-12-09T20:48:36.338 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/google/guava/guava-parent/18.0/guava-parent-18.0.pom 2019-12-09T20:48:36.436 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.11.0/xercesImpl-2.11.0.pom 2019-12-09T20:48:36.436 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.3.3/httpmime-4.3.3.pom 2019-12-09T20:48:36.436 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/nekohtml/nekohtml/1.9.21/nekohtml-1.9.21.pom 2019-12-09T20:48:36.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.pom 2019-12-09T20:48:36.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/xalan/xalan/2.7.1/xalan-2.7.1.pom 2019-12-09T20:48:36.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/htmlunit/htmlunit-core-js/2.15/htmlunit-core-js-2.15.pom 2019-12-09T20:48:36.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.pom 2019-12-09T20:48:36.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-websocket/8.1.15.v20140411/jetty-websocket-8.1.15.v20140411.pom 2019-12-09T20:48:36.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/9/commons-parent-9.pom 2019-12-09T20:48:36.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcomponents-client/4.3.3/httpcomponents-client-4.3.3.pom 2019-12-09T20:48:36.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/apache/4/apache-4.pom 2019-12-09T20:48:36.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/cssparser/cssparser/0.9.14/cssparser-0.9.14.pom 2019-12-09T20:48:36.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-parent/33/commons-parent-33.pom 2019-12-09T20:48:36.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-project/8.1.15.v20140411/jetty-project-8.1.15.v20140411.pom 2019-12-09T20:48:36.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/project/7/project-7.pom 2019-12-09T20:48:36.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-parent/20/jetty-parent-20.pom 2019-12-09T20:48:36.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/xalan/serializer/2.7.1/serializer-2.7.1.pom 2019-12-09T20:48:36.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/xml-apis/xml-apis/1.4.01/xml-apis-1.4.01.pom 2019-12-09T20:48:36.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/w3c/css/sac/1.3/sac-1.3.pom 2019-12-09T20:48:36.535 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/8.1.15.v20140411/jetty-util-8.1.15.v20140411.pom 2019-12-09T20:48:36.536 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-http/8.1.15.v20140411/jetty-http-8.1.15.v20140411.pom 2019-12-09T20:48:36.536 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/8.1.15.v20140411/jetty-io-8.1.15.v20140411.pom 2019-12-09T20:48:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk/1.11.549/aws-java-sdk-1.11.549.jar 2019-12-09T20:48:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-s3control/1.11.549/aws-java-sdk-s3control-1.11.549.jar 2019-12-09T20:48:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudwatchmetrics/1.11.549/aws-java-sdk-cloudwatchmetrics-1.11.549.jar 2019-12-09T20:48:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-glacier/1.11.549/aws-java-sdk-glacier-1.11.549.jar 2019-12-09T20:48:37.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-s3/1.11.549/aws-java-sdk-s3-1.11.549.jar 2019-12-09T20:48:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dynamodb/1.11.549/aws-java-sdk-dynamodb-1.11.549.jar 2019-12-09T20:48:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sns/1.11.549/aws-java-sdk-sns-1.11.549.jar 2019-12-09T20:48:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sqs/1.11.549/aws-java-sdk-sqs-1.11.549.jar 2019-12-09T20:48:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-managedblockchain/1.11.549/aws-java-sdk-managedblockchain-1.11.549.jar 2019-12-09T20:48:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-textract/1.11.549/aws-java-sdk-textract-1.11.549.jar 2019-12-09T20:48:37.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-worklink/1.11.549/aws-java-sdk-worklink-1.11.549.jar 2019-12-09T20:48:37.436 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-backup/1.11.549/aws-java-sdk-backup-1.11.549.jar 2019-12-09T20:48:37.436 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-apigatewayv2/1.11.549/aws-java-sdk-apigatewayv2-1.11.549.jar 2019-12-09T20:48:37.436 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-docdb/1.11.549/aws-java-sdk-docdb-1.11.549.jar 2019-12-09T20:48:37.436 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-apigatewaymanagementapi/1.11.549/aws-java-sdk-apigatewaymanagementapi-1.11.549.jar 2019-12-09T20:48:37.436 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-licensemanager/1.11.549/aws-java-sdk-licensemanager-1.11.549.jar 2019-12-09T20:48:37.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediaconnect/1.11.549/aws-java-sdk-mediaconnect-1.11.549.jar 2019-12-09T20:48:37.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kafka/1.11.549/aws-java-sdk-kafka-1.11.549.jar 2019-12-09T20:48:37.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appmesh/1.11.549/aws-java-sdk-appmesh-1.11.549.jar 2019-12-09T20:48:37.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-comprehendmedical/1.11.549/aws-java-sdk-comprehendmedical-1.11.549.jar 2019-12-09T20:48:37.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesisanalyticsv2/1.11.549/aws-java-sdk-kinesisanalyticsv2-1.11.549.jar 2019-12-09T20:48:37.437 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-fsx/1.11.549/aws-java-sdk-fsx-1.11.549.jar 2019-12-09T20:48:37.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-securityhub/1.11.549/aws-java-sdk-securityhub-1.11.549.jar 2019-12-09T20:48:37.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-globalaccelerator/1.11.549/aws-java-sdk-globalaccelerator-1.11.549.jar 2019-12-09T20:48:37.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-transfer/1.11.549/aws-java-sdk-transfer-1.11.549.jar 2019-12-09T20:48:37.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-datasync/1.11.549/aws-java-sdk-datasync-1.11.549.jar 2019-12-09T20:48:37.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-amplify/1.11.549/aws-java-sdk-amplify-1.11.549.jar 2019-12-09T20:48:37.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-robomaker/1.11.549/aws-java-sdk-robomaker-1.11.549.jar 2019-12-09T20:48:37.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-quicksight/1.11.549/aws-java-sdk-quicksight-1.11.549.jar 2019-12-09T20:48:37.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rdsdata/1.11.549/aws-java-sdk-rdsdata-1.11.549.jar 2019-12-09T20:48:37.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-route53resolver/1.11.549/aws-java-sdk-route53resolver-1.11.549.jar 2019-12-09T20:48:37.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpointsmsvoice/1.11.549/aws-java-sdk-pinpointsmsvoice-1.11.549.jar 2019-12-09T20:48:37.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ram/1.11.549/aws-java-sdk-ram-1.11.549.jar 2019-12-09T20:48:37.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-chime/1.11.549/aws-java-sdk-chime-1.11.549.jar 2019-12-09T20:48:37.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-signer/1.11.549/aws-java-sdk-signer-1.11.549.jar 2019-12-09T20:48:37.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpointemail/1.11.549/aws-java-sdk-pinpointemail-1.11.549.jar 2019-12-09T20:48:37.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dlm/1.11.549/aws-java-sdk-dlm-1.11.549.jar 2019-12-09T20:48:37.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-macie/1.11.549/aws-java-sdk-macie-1.11.549.jar 2019-12-09T20:48:37.542 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediatailor/1.11.549/aws-java-sdk-mediatailor-1.11.549.jar 2019-12-09T20:48:37.543 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-eks/1.11.549/aws-java-sdk-eks-1.11.549.jar 2019-12-09T20:48:37.543 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pi/1.11.549/aws-java-sdk-pi-1.11.549.jar 2019-12-09T20:48:37.543 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-neptune/1.11.549/aws-java-sdk-neptune-1.11.549.jar 2019-12-09T20:48:37.543 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot1clickprojects/1.11.549/aws-java-sdk-iot1clickprojects-1.11.549.jar 2019-12-09T20:48:37.543 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot1clickdevices/1.11.549/aws-java-sdk-iot1clickdevices-1.11.549.jar 2019-12-09T20:48:37.543 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iotanalytics/1.11.549/aws-java-sdk-iotanalytics-1.11.549.jar 2019-12-09T20:48:37.544 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-acmpca/1.11.549/aws-java-sdk-acmpca-1.11.549.jar 2019-12-09T20:48:37.544 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-secretsmanager/1.11.549/aws-java-sdk-secretsmanager-1.11.549.jar 2019-12-09T20:48:37.544 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-fms/1.11.549/aws-java-sdk-fms-1.11.549.jar 2019-12-09T20:48:37.544 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-transcribe/1.11.549/aws-java-sdk-transcribe-1.11.549.jar 2019-12-09T20:48:37.544 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-connect/1.11.549/aws-java-sdk-connect-1.11.549.jar 2019-12-09T20:48:37.544 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-autoscalingplans/1.11.549/aws-java-sdk-autoscalingplans-1.11.549.jar 2019-12-09T20:48:37.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workmail/1.11.549/aws-java-sdk-workmail-1.11.549.jar 2019-12-09T20:48:37.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servicediscovery/1.11.549/aws-java-sdk-servicediscovery-1.11.549.jar 2019-12-09T20:48:37.639 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloud9/1.11.549/aws-java-sdk-cloud9-1.11.549.jar 2019-12-09T20:48:37.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-serverlessapplicationrepository/1.11.549/aws-java-sdk-serverlessapplicationrepository-1.11.549.jar 2019-12-09T20:48:37.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-resourcegroups/1.11.549/aws-java-sdk-resourcegroups-1.11.549.jar 2019-12-09T20:48:37.640 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-comprehend/1.11.549/aws-java-sdk-comprehend-1.11.549.jar 2019-12-09T20:48:37.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-translate/1.11.549/aws-java-sdk-translate-1.11.549.jar 2019-12-09T20:48:37.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sagemaker/1.11.549/aws-java-sdk-sagemaker-1.11.549.jar 2019-12-09T20:48:37.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iotjobsdataplane/1.11.549/aws-java-sdk-iotjobsdataplane-1.11.549.jar 2019-12-09T20:48:37.641 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesisvideo/1.11.549/aws-java-sdk-kinesisvideo-1.11.549.jar 2019-12-09T20:48:37.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sagemakerruntime/1.11.549/aws-java-sdk-sagemakerruntime-1.11.549.jar 2019-12-09T20:48:37.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appsync/1.11.549/aws-java-sdk-appsync-1.11.549.jar 2019-12-09T20:48:37.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-alexaforbusiness/1.11.549/aws-java-sdk-alexaforbusiness-1.11.549.jar 2019-12-09T20:48:37.642 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mq/1.11.549/aws-java-sdk-mq-1.11.549.jar 2019-12-09T20:48:37.643 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediastore/1.11.549/aws-java-sdk-mediastore-1.11.549.jar 2019-12-09T20:48:37.643 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-guardduty/1.11.549/aws-java-sdk-guardduty-1.11.549.jar 2019-12-09T20:48:37.643 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediaconvert/1.11.549/aws-java-sdk-mediaconvert-1.11.549.jar 2019-12-09T20:48:37.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediastoredata/1.11.549/aws-java-sdk-mediastoredata-1.11.549.jar 2019-12-09T20:48:37.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-costexplorer/1.11.549/aws-java-sdk-costexplorer-1.11.549.jar 2019-12-09T20:48:37.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mediapackage/1.11.549/aws-java-sdk-mediapackage-1.11.549.jar 2019-12-09T20:48:37.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mobile/1.11.549/aws-java-sdk-mobile-1.11.549.jar 2019-12-09T20:48:37.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-medialive/1.11.549/aws-java-sdk-medialive-1.11.549.jar 2019-12-09T20:48:37.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pricing/1.11.549/aws-java-sdk-pricing-1.11.549.jar 2019-12-09T20:48:37.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudhsmv2/1.11.549/aws-java-sdk-cloudhsmv2-1.11.549.jar 2019-12-09T20:48:37.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-migrationhub/1.11.549/aws-java-sdk-migrationhub-1.11.549.jar 2019-12-09T20:48:37.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-glue/1.11.549/aws-java-sdk-glue-1.11.549.jar 2019-12-09T20:48:37.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dax/1.11.549/aws-java-sdk-dax-1.11.549.jar 2019-12-09T20:48:37.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-greengrass/1.11.549/aws-java-sdk-greengrass-1.11.549.jar 2019-12-09T20:48:37.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplaceentitlement/1.11.549/aws-java-sdk-marketplaceentitlement-1.11.549.jar 2019-12-09T20:48:37.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codestar/1.11.549/aws-java-sdk-codestar-1.11.549.jar 2019-12-09T20:48:37.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-athena/1.11.549/aws-java-sdk-athena-1.11.549.jar 2019-12-09T20:48:37.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lexmodelbuilding/1.11.549/aws-java-sdk-lexmodelbuilding-1.11.549.jar 2019-12-09T20:48:37.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-resourcegroupstaggingapi/1.11.549/aws-java-sdk-resourcegroupstaggingapi-1.11.549.jar 2019-12-09T20:48:37.837 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-pinpoint/1.11.549/aws-java-sdk-pinpoint-1.11.549.jar 2019-12-09T20:48:37.837 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-opsworkscm/1.11.549/aws-java-sdk-opsworkscm-1.11.549.jar 2019-12-09T20:48:37.837 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-xray/1.11.549/aws-java-sdk-xray-1.11.549.jar 2019-12-09T20:48:37.838 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-support/1.11.549/aws-java-sdk-support-1.11.549.jar 2019-12-09T20:48:37.838 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-simpledb/1.11.549/aws-java-sdk-simpledb-1.11.549.jar 2019-12-09T20:48:37.838 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-swf-libraries/1.11.22/aws-java-sdk-swf-libraries-1.11.22.jar 2019-12-09T20:48:37.838 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servermigration/1.11.549/aws-java-sdk-servermigration-1.11.549.jar 2019-12-09T20:48:37.839 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-servicecatalog/1.11.549/aws-java-sdk-servicecatalog-1.11.549.jar 2019-12-09T20:48:37.839 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-simpleworkflow/1.11.549/aws-java-sdk-simpleworkflow-1.11.549.jar 2019-12-09T20:48:37.839 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-route53/1.11.549/aws-java-sdk-route53-1.11.549.jar 2019-12-09T20:48:37.839 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-importexport/1.11.549/aws-java-sdk-importexport-1.11.549.jar 2019-12-09T20:48:37.839 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-storagegateway/1.11.549/aws-java-sdk-storagegateway-1.11.549.jar 2019-12-09T20:48:37.840 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-sts/1.11.549/aws-java-sdk-sts-1.11.549.jar 2019-12-09T20:48:37.840 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rds/1.11.549/aws-java-sdk-rds-1.11.549.jar 2019-12-09T20:48:37.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-redshift/1.11.549/aws-java-sdk-redshift-1.11.549.jar 2019-12-09T20:48:37.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticbeanstalk/1.11.549/aws-java-sdk-elasticbeanstalk-1.11.549.jar 2019-12-09T20:48:37.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-datapipeline/1.11.549/aws-java-sdk-datapipeline-1.11.549.jar 2019-12-09T20:48:37.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iam/1.11.549/aws-java-sdk-iam-1.11.549.jar 2019-12-09T20:48:37.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticloadbalancing/1.11.549/aws-java-sdk-elasticloadbalancing-1.11.549.jar 2019-12-09T20:48:37.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticache/1.11.549/aws-java-sdk-elasticache-1.11.549.jar 2019-12-09T20:48:37.938 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elastictranscoder/1.11.549/aws-java-sdk-elastictranscoder-1.11.549.jar 2019-12-09T20:48:37.938 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticloadbalancingv2/1.11.549/aws-java-sdk-elasticloadbalancingv2-1.11.549.jar 2019-12-09T20:48:37.938 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-budgets/1.11.549/aws-java-sdk-budgets-1.11.549.jar 2019-12-09T20:48:37.938 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-emr/1.11.549/aws-java-sdk-emr-1.11.549.jar 2019-12-09T20:48:37.938 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ec2/1.11.549/aws-java-sdk-ec2-1.11.549.jar 2019-12-09T20:48:37.938 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudtrail/1.11.549/aws-java-sdk-cloudtrail-1.11.549.jar 2019-12-09T20:48:38.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-logs/1.11.549/aws-java-sdk-logs-1.11.549.jar 2019-12-09T20:48:38.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudwatch/1.11.549/aws-java-sdk-cloudwatch-1.11.549.jar 2019-12-09T20:48:38.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-events/1.11.549/aws-java-sdk-events-1.11.549.jar 2019-12-09T20:48:38.036 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitoidentity/1.11.549/aws-java-sdk-cognitoidentity-1.11.549.jar 2019-12-09T20:48:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitosync/1.11.549/aws-java-sdk-cognitosync-1.11.549.jar 2019-12-09T20:48:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-clouddirectory/1.11.549/aws-java-sdk-clouddirectory-1.11.549.jar 2019-12-09T20:48:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudformation/1.11.549/aws-java-sdk-cloudformation-1.11.549.jar 2019-12-09T20:48:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-directconnect/1.11.549/aws-java-sdk-directconnect-1.11.549.jar 2019-12-09T20:48:38.037 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudfront/1.11.549/aws-java-sdk-cloudfront-1.11.549.jar 2019-12-09T20:48:38.038 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kinesis/1.11.549/aws-java-sdk-kinesis-1.11.549.jar 2019-12-09T20:48:38.038 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ses/1.11.549/aws-java-sdk-ses-1.11.549.jar 2019-12-09T20:48:38.136 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-opsworks/1.11.549/aws-java-sdk-opsworks-1.11.549.jar 2019-12-09T20:48:38.136 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-autoscaling/1.11.549/aws-java-sdk-autoscaling-1.11.549.jar 2019-12-09T20:48:38.137 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudsearch/1.11.549/aws-java-sdk-cloudsearch-1.11.549.jar 2019-12-09T20:48:38.137 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codedeploy/1.11.549/aws-java-sdk-codedeploy-1.11.549.jar 2019-12-09T20:48:38.137 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lambda/1.11.549/aws-java-sdk-lambda-1.11.549.jar 2019-12-09T20:48:38.137 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codepipeline/1.11.549/aws-java-sdk-codepipeline-1.11.549.jar 2019-12-09T20:48:38.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-config/1.11.549/aws-java-sdk-config-1.11.549.jar 2019-12-09T20:48:38.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ecr/1.11.549/aws-java-sdk-ecr-1.11.549.jar 2019-12-09T20:48:38.138 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-kms/1.11.549/aws-java-sdk-kms-1.11.549.jar 2019-12-09T20:48:38.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cloudhsm/1.11.549/aws-java-sdk-cloudhsm-1.11.549.jar 2019-12-09T20:48:38.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ecs/1.11.549/aws-java-sdk-ecs-1.11.549.jar 2019-12-09T20:48:38.236 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workspaces/1.11.549/aws-java-sdk-workspaces-1.11.549.jar 2019-12-09T20:48:38.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-efs/1.11.549/aws-java-sdk-efs-1.11.549.jar 2019-12-09T20:48:38.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codecommit/1.11.549/aws-java-sdk-codecommit-1.11.549.jar 2019-12-09T20:48:38.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-directory/1.11.549/aws-java-sdk-directory-1.11.549.jar 2019-12-09T20:48:38.237 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-machinelearning/1.11.549/aws-java-sdk-machinelearning-1.11.549.jar 2019-12-09T20:48:38.238 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-devicefarm/1.11.549/aws-java-sdk-devicefarm-1.11.549.jar 2019-12-09T20:48:38.238 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-ssm/1.11.549/aws-java-sdk-ssm-1.11.549.jar 2019-12-09T20:48:38.238 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-elasticsearch/1.11.549/aws-java-sdk-elasticsearch-1.11.549.jar 2019-12-09T20:48:38.238 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplacecommerceanalytics/1.11.549/aws-java-sdk-marketplacecommerceanalytics-1.11.549.jar 2019-12-09T20:48:38.336 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-inspector/1.11.549/aws-java-sdk-inspector-1.11.549.jar 2019-12-09T20:48:38.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-waf/1.11.549/aws-java-sdk-waf-1.11.549.jar 2019-12-09T20:48:38.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-acm/1.11.549/aws-java-sdk-acm-1.11.549.jar 2019-12-09T20:48:38.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-api-gateway/1.11.549/aws-java-sdk-api-gateway-1.11.549.jar 2019-12-09T20:48:38.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-iot/1.11.549/aws-java-sdk-iot-1.11.549.jar 2019-12-09T20:48:38.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-dms/1.11.549/aws-java-sdk-dms-1.11.549.jar 2019-12-09T20:48:38.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-gamelift/1.11.549/aws-java-sdk-gamelift-1.11.549.jar 2019-12-09T20:48:38.337 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-marketplacemeteringservice/1.11.549/aws-java-sdk-marketplacemeteringservice-1.11.549.jar 2019-12-09T20:48:38.338 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-cognitoidp/1.11.549/aws-java-sdk-cognitoidp-1.11.549.jar 2019-12-09T20:48:38.338 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-discovery/1.11.549/aws-java-sdk-discovery-1.11.549.jar 2019-12-09T20:48:38.338 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-applicationautoscaling/1.11.549/aws-java-sdk-applicationautoscaling-1.11.549.jar 2019-12-09T20:48:38.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-polly/1.11.549/aws-java-sdk-polly-1.11.549.jar 2019-12-09T20:48:38.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-snowball/1.11.549/aws-java-sdk-snowball-1.11.549.jar 2019-12-09T20:48:38.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-health/1.11.549/aws-java-sdk-health-1.11.549.jar 2019-12-09T20:48:38.438 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-stepfunctions/1.11.549/aws-java-sdk-stepfunctions-1.11.549.jar 2019-12-09T20:48:38.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-rekognition/1.11.549/aws-java-sdk-rekognition-1.11.549.jar 2019-12-09T20:48:38.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-appstream/1.11.549/aws-java-sdk-appstream-1.11.549.jar 2019-12-09T20:48:38.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lightsail/1.11.549/aws-java-sdk-lightsail-1.11.549.jar 2019-12-09T20:48:38.439 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-costandusagereport/1.11.549/aws-java-sdk-costandusagereport-1.11.549.jar 2019-12-09T20:48:38.440 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-codebuild/1.11.549/aws-java-sdk-codebuild-1.11.549.jar 2019-12-09T20:48:38.440 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-shield/1.11.549/aws-java-sdk-shield-1.11.549.jar 2019-12-09T20:48:38.440 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-batch/1.11.549/aws-java-sdk-batch-1.11.549.jar 2019-12-09T20:48:38.441 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-mechanicalturkrequester/1.11.549/aws-java-sdk-mechanicalturkrequester-1.11.549.jar 2019-12-09T20:48:38.536 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-organizations/1.11.549/aws-java-sdk-organizations-1.11.549.jar 2019-12-09T20:48:38.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-lex/1.11.549/aws-java-sdk-lex-1.11.549.jar 2019-12-09T20:48:38.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.549/aws-java-sdk-core-1.11.549.jar 2019-12-09T20:48:38.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-workdocs/1.11.549/aws-java-sdk-workdocs-1.11.549.jar 2019-12-09T20:48:38.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-server/2.44.0/selenium-server-2.44.0.jar 2019-12-09T20:48:38.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/log4j/log4j/1.2.17/log4j-1.2.17.jar 2019-12-09T20:48:38.537 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/testng/testng/6.1.1/testng-6.1.1.jar 2019-12-09T20:48:38.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-java/2.44.0/selenium-java-2.44.0.jar 2019-12-09T20:48:38.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-htmlunit-driver/2.44.0/selenium-htmlunit-driver-2.44.0.jar 2019-12-09T20:48:38.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-chrome-driver/2.44.0/selenium-chrome-driver-2.44.0.jar 2019-12-09T20:48:38.538 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-ie-driver/2.44.0/selenium-ie-driver-2.44.0.jar 2019-12-09T20:48:38.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-safari-driver/2.44.0/selenium-safari-driver-2.44.0.jar 2019-12-09T20:48:38.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpmime/4.3.3/httpmime-4.3.3.jar 2019-12-09T20:48:38.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-remote-driver/2.44.0/selenium-remote-driver-2.44.0.jar 2019-12-09T20:48:38.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-support/2.44.0/selenium-support-2.44.0.jar 2019-12-09T20:48:38.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpclient/4.5.5/httpclient-4.5.5.jar 2019-12-09T20:48:38.539 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/jmespath-java/1.11.549/jmespath-java-1.11.549.jar 2019-12-09T20:48:38.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-logging/commons-logging/1.2/commons-logging-1.2.jar 2019-12-09T20:48:38.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/software/amazon/ion/ion-java/1.0.2/ion-java-1.0.2.jar 2019-12-09T20:48:38.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/htmlunit/htmlunit/2.15/htmlunit-2.15.jar 2019-12-09T20:48:38.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.6.7/jackson-dataformat-cbor-2.6.7.jar 2019-12-09T20:48:38.540 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.6.7.2/jackson-databind-2.6.7.2.jar 2019-12-09T20:48:38.541 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/joda-time/joda-time/2.8.1/joda-time-2.8.1.jar 2019-12-09T20:48:38.541 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-firefox-driver/2.44.0/selenium-firefox-driver-2.44.0.jar 2019-12-09T20:48:38.542 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.48/bcpkix-jdk15on-1.48.jar 2019-12-09T20:48:38.636 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/bouncycastle/bcprov-jdk15on/1.48/bcprov-jdk15on-1.48.jar 2019-12-09T20:48:38.637 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/mx4j/mx4j-tools/3.0.1/mx4j-tools-3.0.1.jar 2019-12-09T20:48:38.637 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/mortbay/jetty/servlet-api-2.5/6.1.9/servlet-api-2.5-6.1.9.jar 2019-12-09T20:48:38.637 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/jetty-repacked/7.6.1/jetty-repacked-7.6.1.jar 2019-12-09T20:48:38.637 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/amazonaws/aws-java-sdk-models/1.11.549/aws-java-sdk-models-1.11.549.jar 2019-12-09T20:48:38.637 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/jetty-rc-repacked/5/jetty-rc-repacked-5.jar 2019-12-09T20:48:38.637 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/jcip/jcip-annotations/1.0/jcip-annotations-1.0.jar 2019-12-09T20:48:38.638 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-codec/commons-codec/1.10/commons-codec-1.10.jar 2019-12-09T20:48:38.638 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/yaml/snakeyaml/1.8/snakeyaml-1.8.jar 2019-12-09T20:48:38.638 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/junit/junit/3.8.1/junit-3.8.1.jar 2019-12-09T20:48:38.736 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/beanshell/bsh/2.0b4/bsh-2.0b4.jar 2019-12-09T20:48:38.736 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/beust/jcommander/1.12/jcommander-1.12.jar 2019-12-09T20:48:38.736 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/httpcomponents/httpcore/4.4.9/httpcore-4.4.9.jar 2019-12-09T20:48:38.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/webbitserver/webbit/0.4.14/webbit-0.4.14.jar 2019-12-09T20:48:38.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-codec/4.1.17.Final/netty-codec-4.1.17.Final.jar 2019-12-09T20:48:38.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.6.0/jackson-annotations-2.6.0.jar 2019-12-09T20:48:38.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-handler/4.1.17.Final/netty-handler-4.1.17.Final.jar 2019-12-09T20:48:38.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-transport/4.1.17.Final/netty-transport-4.1.17.Final.jar 2019-12-09T20:48:38.737 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.6.7/jackson-core-2.6.7.jar 2019-12-09T20:48:38.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-codec-http/4.1.17.Final/netty-codec-http-4.1.17.Final.jar 2019-12-09T20:48:38.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-buffer/4.1.17.Final/netty-buffer-4.1.17.Final.jar 2019-12-09T20:48:38.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-io/commons-io/2.4/commons-io-2.4.jar 2019-12-09T20:48:38.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/java/dev/jna/jna/3.4.0/jna-3.4.0.jar 2019-12-09T20:48:38.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-exec/1.1/commons-exec-1.1.jar 2019-12-09T20:48:38.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/java/dev/jna/platform/3.4.0/platform-3.4.0.jar 2019-12-09T20:48:38.738 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-resolver/4.1.17.Final/netty-resolver-4.1.17.Final.jar 2019-12-09T20:48:38.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty-common/4.1.17.Final/netty-common-4.1.17.Final.jar 2019-12-09T20:48:38.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/seleniumhq/selenium/selenium-api/2.44.0/selenium-api-2.44.0.jar 2019-12-09T20:48:38.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/cglib/cglib-nodep/2.1_3/cglib-nodep-2.1_3.jar 2019-12-09T20:48:38.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/google/code/gson/gson/2.3/gson-2.3.jar 2019-12-09T20:48:38.739 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/io/netty/netty/3.5.2.Final/netty-3.5.2.Final.jar 2019-12-09T20:48:38.836 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/com/google/guava/guava/18.0/guava-18.0.jar 2019-12-09T20:48:38.836 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar 2019-12-09T20:48:38.836 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/xalan/xalan/2.7.1/xalan-2.7.1.jar 2019-12-09T20:48:38.836 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/htmlunit/htmlunit-core-js/2.15/htmlunit-core-js-2.15.jar 2019-12-09T20:48:38.837 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/xerces/xercesImpl/2.11.0/xercesImpl-2.11.0.jar 2019-12-09T20:48:38.837 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/apache/commons/commons-lang3/3.3.2/commons-lang3-3.3.2.jar 2019-12-09T20:48:38.837 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/nekohtml/nekohtml/1.9.21/nekohtml-1.9.21.jar 2019-12-09T20:48:38.837 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/net/sourceforge/cssparser/cssparser/0.9.14/cssparser-0.9.14.jar 2019-12-09T20:48:38.837 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-websocket/8.1.15.v20140411/jetty-websocket-8.1.15.v20140411.jar 2019-12-09T20:48:38.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-http/8.1.15.v20140411/jetty-http-8.1.15.v20140411.jar 2019-12-09T20:48:38.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/w3c/css/sac/1.3/sac-1.3.jar 2019-12-09T20:48:38.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/xalan/serializer/2.7.1/serializer-2.7.1.jar 2019-12-09T20:48:38.936 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/xml-apis/xml-apis/1.4.01/xml-apis-1.4.01.jar 2019-12-09T20:48:38.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-io/8.1.15.v20140411/jetty-io-8.1.15.v20140411.jar 2019-12-09T20:48:38.937 INFO:teuthology.orchestra.run.smithi067.stdout:Download https://repo.maven.apache.org/maven2/org/eclipse/jetty/jetty-util/8.1.15.v20140411/jetty-util-8.1.15.v20140411.jar 2019-12-09T20:48:47.536 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :compileJava 2019-12-09T20:48:47.537 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :processResources 2019-12-09T20:48:47.537 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :classes 2019-12-09T20:48:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:48.039 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :compileTestJava 2019-12-09T20:48:48.039 INFO:teuthology.orchestra.run.smithi067.stdout:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2019-12-09T20:48:48.039 INFO:teuthology.orchestra.run.smithi067.stdout:Note: Recompile with -Xlint:deprecation for details. 2019-12-09T20:48:48.039 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:48.040 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :processTestResources NO-SOURCE 2019-12-09T20:48:48.040 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :testClasses 2019-12-09T20:48:55.437 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:55.437 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :test 2019-12-09T20:48:55.438 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:48:55.438 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 STARTED 2019-12-09T20:49:38.240 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:49:38.240 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 PASSED 2019-12-09T20:49:44.140 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:49:44.141 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 STARTED 2019-12-09T20:49:48.340 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:49:48.341 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 PASSED 2019-12-09T20:49:55.541 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:49:55.541 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 STARTED 2019-12-09T20:50:01.195 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:22.268 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:50:01.198 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:22.271 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:50:01.241 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:16:22.420 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:50:01.245 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:16:22.425 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:50:01.252 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:22.325 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:50:01.253 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:22.327 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:50:01.411 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:22.484 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:50:01.435 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:22.509 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:50:01.476 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:16:22.656 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:50:19.243 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:50:19.243 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 PASSED 2019-12-09T20:50:26.543 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:50:26.544 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 STARTED 2019-12-09T20:50:27.043 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:50:27.044 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 PASSED 2019-12-09T20:50:27.844 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:50:27.844 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 STARTED 2019-12-09T20:50:29.743 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:50:29.744 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 PASSED 2019-12-09T20:50:36.544 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:50:36.544 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 STARTED 2019-12-09T20:51:05.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:51:05.446 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 PASSED 2019-12-09T20:51:07.268 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:28.336 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:51:07.295 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:28.364 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:51:07.334 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:28.398 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:51:07.334 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:28.401 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:51:07.357 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:17:28.537 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:51:07.358 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:28.427 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:51:07.368 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:17:28.548 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:51:07.384 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:28.452 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:51:07.492 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:17:28.673 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:51:17.849 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:51:17.849 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 STARTED 2019-12-09T20:55:02.663 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:55:02.691 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 PASSED 2019-12-09T20:55:11.251 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:32.304 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:55:11.287 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:32.340 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:55:11.306 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:21:32.491 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:55:11.327 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:21:32.512 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:55:11.337 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:32.383 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:55:11.337 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:32.385 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:55:11.443 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:32.496 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:55:11.476 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:32.529 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:55:11.480 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:21:32.665 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:55:14.562 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:55:14.562 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 STARTED 2019-12-09T20:55:16.762 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:55:16.762 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 PASSED 2019-12-09T20:55:24.062 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:55:24.062 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 STARTED 2019-12-09T20:55:28.462 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:55:28.463 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 PASSED 2019-12-09T20:55:39.362 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:55:39.363 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 STARTED 2019-12-09T20:56:08.062 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:56:08.064 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 PASSED 2019-12-09T20:56:14.764 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:56:14.765 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 STARTED 2019-12-09T20:56:15.074 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:56:15.074 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 PASSED 2019-12-09T20:56:16.063 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:56:16.063 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 STARTED 2019-12-09T20:56:38.963 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:56:38.963 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 PASSED 2019-12-09T20:56:44.763 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:56:44.763 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 STARTED 2019-12-09T20:57:05.765 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:57:05.765 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 PASSED 2019-12-09T20:57:16.363 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:57:16.363 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 STARTED 2019-12-09T20:57:34.463 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:57:34.463 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 PASSED 2019-12-09T20:57:41.263 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T20:57:41.263 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 STARTED 2019-12-09T20:59:28.685 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:49.736 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:59:28.729 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:49.779 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:59:28.756 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:49.806 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:59:28.757 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:49.808 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:59:28.759 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:25:49.945 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:59:28.762 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:25:49.949 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T20:59:28.859 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:49.909 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T20:59:28.898 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:49.949 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T20:59:28.918 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:25:50.105 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:05:34.565 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:55.613 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:05:34.575 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:55.623 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:05:34.578 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:31:55.747 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:05:34.584 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:55.632 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:05:34.585 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:31:55.754 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:05:34.591 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:55.639 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:05:34.802 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:55.850 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:05:34.803 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:31:55.973 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:05:34.846 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:55.894 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:06:21.072 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:06:21.072 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 PASSED 2019-12-09T21:06:26.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:06:26.568 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 STARTED 2019-12-09T21:07:27.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:07:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 PASSED 2019-12-09T21:07:35.175 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:07:35.180 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 STARTED 2019-12-09T21:08:13.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:08:13.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 PASSED 2019-12-09T21:08:20.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:08:20.768 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 STARTED 2019-12-09T21:08:28.369 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:08:28.369 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 PASSED 2019-12-09T21:08:37.968 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:08:37.968 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 STARTED 2019-12-09T21:08:41.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:08:41.068 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 PASSED 2019-12-09T21:08:45.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:08:45.069 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 STARTED 2019-12-09T21:08:47.570 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:08:47.570 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 PASSED 2019-12-09T21:08:55.669 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:08:55.669 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 STARTED 2019-12-09T21:09:00.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:00.369 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 PASSED 2019-12-09T21:09:05.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:05.369 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 STARTED 2019-12-09T21:09:09.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:09.369 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 PASSED 2019-12-09T21:09:16.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:16.169 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 STARTED 2019-12-09T21:09:18.668 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:18.669 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 PASSED 2019-12-09T21:09:26.169 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:26.169 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 STARTED 2019-12-09T21:09:29.969 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:29.969 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 PASSED 2019-12-09T21:09:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:35.269 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 STARTED 2019-12-09T21:09:37.669 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:37.669 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 PASSED 2019-12-09T21:09:44.770 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:09:44.770 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 STARTED 2019-12-09T21:10:43.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:10:43.369 INFO:teuthology.orchestra.run.smithi067.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 PASSED 2019-12-09T21:10:45.757 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:37:06.804 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:10:45.765 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:37:06.812 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:10:45.775 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:37:06.821 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:10:45.781 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:37:06.827 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:10:45.781 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:37:06.943 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:10:45.795 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:37:06.956 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:10:45.933 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:37:06.979 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:10:45.948 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:37:06.994 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:10:45.965 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:37:07.127 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:10:49.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:10:49.469 INFO:teuthology.orchestra.run.smithi067.stdout:---------------------------------------------------------------------- 2019-12-09T21:10:49.469 INFO:teuthology.orchestra.run.smithi067.stdout:| Results: SUCCESS (26 tests, 26 successes, 0 failures, 0 skipped) | 2019-12-09T21:10:49.469 INFO:teuthology.orchestra.run.smithi067.stdout:---------------------------------------------------------------------- 2019-12-09T21:10:49.495 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:10:49.495 INFO:teuthology.orchestra.run.smithi067.stdout:BUILD SUCCESSFUL in 22m 19s 2019-12-09T21:10:49.495 INFO:teuthology.orchestra.run.smithi067.stdout:5 actionable tasks: 4 executed, 1 up-to-date 2019-12-09T21:10:49.813 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:10:49.954 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:49.935+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:49.981 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:49.963+0000 7fbb96767ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:10:49.982 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:49.963+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:49.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:49.972+0000 7fbb96767ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:10:49.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:49.972+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:49.995 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:49.976+0000 7fbb96767ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:10:49.996 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:49.976+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.002 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:49.984+0000 7fbb96767ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:10:50.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.181+0000 7fbb96767ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:10:50.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.181+0000 7fbb96767ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:10:50.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.181+0000 7fbb96767ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:10:50.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.181+0000 7fbb96767ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:10:50.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.181+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.214 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.196+0000 7fbb96767ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:10:50.214 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.196+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.207+0000 7fbb96767ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:10:50.225 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.207+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.237 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.219+0000 7fbb96767ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:10:50.237 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.219+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.227+0000 7fbb96767ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:10:50.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.227+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.254 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.234+0000 7fbb96767ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:10:50.255 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.234+0000 7fbb96767ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:10:50.255 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.234+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.263 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.239+0000 7fbb96767ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:10:50.263 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.239+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.281 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.249+0000 7fbb96767ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:10:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.249+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.257+0000 7fbb96767ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:10:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.257+0000 7fbb96767ec0 20 zonegroup default 2019-12-09T21:10:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.257+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.283 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.263+0000 7fbb96767ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:10:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.263+0000 7fbb96767ec0 10 Cannot find current period zone using local zone 2019-12-09T21:10:50.284 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.263+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.287 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.268+0000 7fbb96767ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:10:50.288 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.268+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.277+0000 7fbb96767ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:10:50.295 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.277+0000 7fbb96767ec0 20 rados->read ofs=0 len=0 2019-12-09T21:10:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.285+0000 7fbb96767ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:10:50.303 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.285+0000 7fbb96767ec0 20 zone default found 2019-12-09T21:10:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.285+0000 7fbb96767ec0 20 started sync module instance, tier type = 2019-12-09T21:10:50.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.285+0000 7fbb96767ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:10:50.745 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.726+0000 7fbb96767ec0 20 add_watcher() i=0 2019-12-09T21:10:50.745 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.726+0000 7fbb96767ec0 20 add_watcher() i=1 2019-12-09T21:10:50.745 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.726+0000 7fbb96767ec0 20 add_watcher() i=2 2019-12-09T21:10:50.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.726+0000 7fbb96767ec0 20 add_watcher() i=3 2019-12-09T21:10:50.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.726+0000 7fbb96767ec0 20 add_watcher() i=4 2019-12-09T21:10:50.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.726+0000 7fbb96767ec0 20 add_watcher() i=5 2019-12-09T21:10:50.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.726+0000 7fbb96767ec0 20 add_watcher() i=6 2019-12-09T21:10:50.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.766+0000 7fbb96767ec0 20 add_watcher() i=7 2019-12-09T21:10:50.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.766+0000 7fbb96767ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:10:50.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.767+0000 7fbad7fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:10:50.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.771+0000 7fbb83190700 0 -- 172.21.15.67:0/152673113 >> v1:172.21.15.67:6789/0 conn(0x557bfcbf7270 legacy=0x557bfcc2d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T21:10:50.845 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:50.827+0000 7fbb83991700 0 -- 172.21.15.67:0/488924207 >> [v2:172.21.15.67:6800/9311,v1:172.21.15.67:6801/9311] conn(0x7fbac4022470 msgr2=0x7fbac4024960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:10:52.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.228+0000 7fbac27fc700 20 reqs_thread_entry: start 2019-12-09T21:10:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.229+0000 7fbabbfff700 20 reqs_thread_entry: start 2019-12-09T21:10:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.229+0000 7fbb96767ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:10:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.229+0000 7fbaba7fc700 20 reqs_thread_entry: start 2019-12-09T21:10:52.248 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.229+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:10:52.291 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.273+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:52.317 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.289+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.315+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:10:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.315+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr', time=2019-12-09T23:08:35.510109+0000, chain.objs.size()=34 2019-12-09T21:10:52.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.315+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.1 2019-12-09T21:10:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.315+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.2 2019-12-09T21:10:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.315+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.3 2019-12-09T21:10:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.315+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.4 2019-12-09T21:10:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.315+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.5 2019-12-09T21:10:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.316+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.6 2019-12-09T21:10:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.316+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.7 2019-12-09T21:10:52.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.316+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.8 2019-12-09T21:10:52.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.316+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.9 2019-12-09T21:10:52.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.316+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.10 2019-12-09T21:10:52.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.316+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.11 2019-12-09T21:10:52.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.316+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.12 2019-12-09T21:10:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.628+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.13 2019-12-09T21:10:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.628+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.14 2019-12-09T21:10:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.628+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.15 2019-12-09T21:10:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.628+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.16 2019-12-09T21:10:52.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.628+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.EGIV0woKCrWM7rzD2mws2VLlmpXmtYl.17 2019-12-09T21:10:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.628+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.POfBoZR1UDy3ryu__3O6Ugw5S6DcpLD.18 2019-12-09T21:10:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.628+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.faFAjmIxGhZ7pQRbr69sH3sBUG9f4i7.19 2019-12-09T21:10:52.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.628+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.CpxLA_-ts7Do3y2Vl9t4A_D60LYHHjw.20 2019-12-09T21:10:52.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.895+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.21 2019-12-09T21:10:52.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.895+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.22 2019-12-09T21:10:52.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.895+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.23 2019-12-09T21:10:53.012 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:52.994+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.24 2019-12-09T21:10:53.042 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.023+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.25 2019-12-09T21:10:53.042 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.023+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.26 2019-12-09T21:10:53.042 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.023+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.27 2019-12-09T21:10:53.042 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.023+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.28 2019-12-09T21:10:53.042 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.023+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.29 2019-12-09T21:10:53.043 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.023+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.30 2019-12-09T21:10:53.043 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.024+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.31 2019-12-09T21:10:53.085 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.067+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.32 2019-12-09T21:10:53.085 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.067+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.33 2019-12-09T21:10:53.223 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.205+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.31__multipart_key1.2~aPuq_90l8YH-TNIFFO0oJ3f4R-IS_vr.34 2019-12-09T21:10:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.526+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:53.680 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.660+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:10:53.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.721+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:53.759 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.741+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:53.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:10:53.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.557', time=2019-12-09T22:57:07.608535+0000, chain.objs.size()=9 2019-12-09T21:10:53.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file.mpg.2~kz8kBQlUlExHNGz8LsesE_rfMP3hj7g.1 2019-12-09T21:10:53.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__shadow_file.mpg.2~kz8kBQlUlExHNGz8LsesE_rfMP3hj7g.1_1 2019-12-09T21:10:53.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file.mpg.2~kz8kBQlUlExHNGz8LsesE_rfMP3hj7g.2 2019-12-09T21:10:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__shadow_file.mpg.2~kz8kBQlUlExHNGz8LsesE_rfMP3hj7g.2_1 2019-12-09T21:10:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file.mpg.2~kz8kBQlUlExHNGz8LsesE_rfMP3hj7g.3 2019-12-09T21:10:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__shadow_file.mpg.2~kz8kBQlUlExHNGz8LsesE_rfMP3hj7g.3_1 2019-12-09T21:10:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file.mpg.2~kz8kBQlUlExHNGz8LsesE_rfMP3hj7g.4 2019-12-09T21:10:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__shadow_file.mpg.2~kz8kBQlUlExHNGz8LsesE_rfMP3hj7g.4_1 2019-12-09T21:10:53.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.753+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file.mpg.2~kz8kBQlUlExHNGz8LsesE_rfMP3hj7g.5 2019-12-09T21:10:53.992 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:53.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:54.108 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.089+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:10:54.151 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.131+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:54.167 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.148+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:10:54.190 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.460', time=2019-12-09T22:54:21.236970+0000, chain.objs.size()=7 2019-12-09T21:10:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~r0eDjsxDL2184y8yCNIk_7b7Gw866uQ.1 2019-12-09T21:10:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~r0eDjsxDL2184y8yCNIk_7b7Gw866uQ.1_1 2019-12-09T21:10:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~r0eDjsxDL2184y8yCNIk_7b7Gw866uQ.1_2 2019-12-09T21:10:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~r0eDjsxDL2184y8yCNIk_7b7Gw866uQ.2 2019-12-09T21:10:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~r0eDjsxDL2184y8yCNIk_7b7Gw866uQ.2_1 2019-12-09T21:10:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~r0eDjsxDL2184y8yCNIk_7b7Gw866uQ.2_2 2019-12-09T21:10:54.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~r0eDjsxDL2184y8yCNIk_7b7Gw866uQ.3 2019-12-09T21:10:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.832', time=2019-12-09T23:10:44.600916+0000, chain.objs.size()=21 2019-12-09T21:10:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.1 2019-12-09T21:10:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.171+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.1_1 2019-12-09T21:10:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.172+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.2 2019-12-09T21:10:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.172+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.2_1 2019-12-09T21:10:54.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.172+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.3 2019-12-09T21:10:54.450 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.432+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.3_1 2019-12-09T21:10:54.451 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.432+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.4 2019-12-09T21:10:54.451 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.432+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.4_1 2019-12-09T21:10:54.451 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.432+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.5 2019-12-09T21:10:54.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.573+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.5_1 2019-12-09T21:10:54.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.573+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.6 2019-12-09T21:10:54.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.573+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.6_1 2019-12-09T21:10:54.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.573+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.7 2019-12-09T21:10:54.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.573+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.7_1 2019-12-09T21:10:54.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.573+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.8 2019-12-09T21:10:54.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.573+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.8_1 2019-12-09T21:10:54.826 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.807+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.9 2019-12-09T21:10:54.826 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.807+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.9_1 2019-12-09T21:10:54.927 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.909+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.10 2019-12-09T21:10:54.927 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:54.909+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__shadow_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.10_1 2019-12-09T21:10:55.081 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.062+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.39__multipart_key1.2~m1AjlcmzQNgtehFTBY37Of9kNBmtbwj.11 2019-12-09T21:10:55.216 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.198+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:55.338 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.320+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:10:55.386 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.365+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:55.407 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.385+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.397+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:10:55.416 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.397+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.472', time=2019-12-09T22:55:04.989844+0000, chain.objs.size()=7 2019-12-09T21:10:55.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.397+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~14xxrYe36qN2lZk5uvW3-4CHx6J21pd.1 2019-12-09T21:10:55.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.397+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~14xxrYe36qN2lZk5uvW3-4CHx6J21pd.1_1 2019-12-09T21:10:55.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.397+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~14xxrYe36qN2lZk5uvW3-4CHx6J21pd.1_2 2019-12-09T21:10:55.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.397+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~14xxrYe36qN2lZk5uvW3-4CHx6J21pd.2 2019-12-09T21:10:55.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.397+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~14xxrYe36qN2lZk5uvW3-4CHx6J21pd.2_1 2019-12-09T21:10:55.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.397+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~14xxrYe36qN2lZk5uvW3-4CHx6J21pd.2_2 2019-12-09T21:10:55.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.397+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~14xxrYe36qN2lZk5uvW3-4CHx6J21pd.3 2019-12-09T21:10:55.694 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.676+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.780+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:10:55.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.859+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:55.898 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.880+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:55.910 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.892+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:55.948 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.930+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:10:55.997 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.979+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:56.013 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:55.994+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:56.027 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.008+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:10:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.008+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.386', time=2019-12-09T22:50:20.991590+0000, chain.objs.size()=9 2019-12-09T21:10:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.008+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.15__multipart_key1.2~d_ChQF5-0D3z9e6tL2hUJtduRth-IaP.1 2019-12-09T21:10:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.008+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.15__shadow_key1.2~d_ChQF5-0D3z9e6tL2hUJtduRth-IaP.1_1 2019-12-09T21:10:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.008+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.15__multipart_key1.2~d_ChQF5-0D3z9e6tL2hUJtduRth-IaP.2 2019-12-09T21:10:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.008+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.15__shadow_key1.2~d_ChQF5-0D3z9e6tL2hUJtduRth-IaP.2_1 2019-12-09T21:10:56.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.009+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.15__multipart_key1.2~d_ChQF5-0D3z9e6tL2hUJtduRth-IaP.3 2019-12-09T21:10:56.029 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.009+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.15__shadow_key1.2~d_ChQF5-0D3z9e6tL2hUJtduRth-IaP.3_1 2019-12-09T21:10:56.029 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.009+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.15__multipart_key1.2~d_ChQF5-0D3z9e6tL2hUJtduRth-IaP.4 2019-12-09T21:10:56.029 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.009+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.15__shadow_key1.2~d_ChQF5-0D3z9e6tL2hUJtduRth-IaP.4_1 2019-12-09T21:10:56.029 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.009+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.15__multipart_key1.2~d_ChQF5-0D3z9e6tL2hUJtduRth-IaP.5 2019-12-09T21:10:56.380 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.359+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:56.498 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.477+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:10:56.543 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.523+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:56.558 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.539+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.551+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:10:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.551+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='2~irpfN19L5el71x7GfPJGXo35bm0LtfL', time=2019-12-09T22:57:37.083590+0000, chain.objs.size()=5 2019-12-09T21:10:56.573 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.551+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.27__multipart_key1.2~irpfN19L5el71x7GfPJGXo35bm0LtfL.7 2019-12-09T21:10:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.551+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.27__shadow_key1.2~irpfN19L5el71x7GfPJGXo35bm0LtfL.7_1 2019-12-09T21:10:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.551+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.27__multipart_key1.2~irpfN19L5el71x7GfPJGXo35bm0LtfL.10 2019-12-09T21:10:56.574 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.551+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.27__shadow_key1.2~irpfN19L5el71x7GfPJGXo35bm0LtfL.10_1 2019-12-09T21:10:56.575 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.551+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.27__multipart_key1.2~irpfN19L5el71x7GfPJGXo35bm0LtfL.13 2019-12-09T21:10:56.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.791+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:56.902 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.884+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:10:56.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.926+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:56.961 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.941+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:56.992 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:10:56.992 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.671', time=2019-12-09T23:08:15.571012+0000, chain.objs.size()=9 2019-12-09T21:10:56.992 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.30__multipart_key1.2~Nma0ctymy54FR2b5m-uRzqtcozM_MOD.1 2019-12-09T21:10:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.30__shadow_key1.2~Nma0ctymy54FR2b5m-uRzqtcozM_MOD.1_1 2019-12-09T21:10:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.30__multipart_key1.2~Nma0ctymy54FR2b5m-uRzqtcozM_MOD.2 2019-12-09T21:10:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.30__shadow_key1.2~Nma0ctymy54FR2b5m-uRzqtcozM_MOD.2_1 2019-12-09T21:10:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.30__multipart_key1.2~Nma0ctymy54FR2b5m-uRzqtcozM_MOD.3 2019-12-09T21:10:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.30__shadow_key1.2~Nma0ctymy54FR2b5m-uRzqtcozM_MOD.3_1 2019-12-09T21:10:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.30__multipart_key1.2~Nma0ctymy54FR2b5m-uRzqtcozM_MOD.4 2019-12-09T21:10:56.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.30__shadow_key1.2~Nma0ctymy54FR2b5m-uRzqtcozM_MOD.4_1 2019-12-09T21:10:56.994 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:56.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.30__multipart_key1.2~Nma0ctymy54FR2b5m-uRzqtcozM_MOD.5 2019-12-09T21:10:57.305 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.287+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:57.481 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.462+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:10:57.529 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.511+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:57.545 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.527+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:57.555 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.535+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:57.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.584+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:10:57.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.627+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.643+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:57.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:10:57.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.528', time=2019-12-09T22:56:41.176428+0000, chain.objs.size()=9 2019-12-09T21:10:57.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.25__multipart_key1.2~xhrerpYgmG5TDcr_Ugh6EWkCM62JIId.1 2019-12-09T21:10:57.677 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.25__shadow_key1.2~xhrerpYgmG5TDcr_Ugh6EWkCM62JIId.1_1 2019-12-09T21:10:57.677 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.25__multipart_key1.2~xhrerpYgmG5TDcr_Ugh6EWkCM62JIId.2 2019-12-09T21:10:57.677 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.25__shadow_key1.2~xhrerpYgmG5TDcr_Ugh6EWkCM62JIId.2_1 2019-12-09T21:10:57.677 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.25__multipart_key1.2~xhrerpYgmG5TDcr_Ugh6EWkCM62JIId.3 2019-12-09T21:10:57.677 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.25__shadow_key1.2~xhrerpYgmG5TDcr_Ugh6EWkCM62JIId.3_1 2019-12-09T21:10:57.677 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.25__multipart_key1.2~xhrerpYgmG5TDcr_Ugh6EWkCM62JIId.4 2019-12-09T21:10:57.678 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.25__shadow_key1.2~xhrerpYgmG5TDcr_Ugh6EWkCM62JIId.4_1 2019-12-09T21:10:57.678 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:57.658+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.25__multipart_key1.2~xhrerpYgmG5TDcr_Ugh6EWkCM62JIId.5 2019-12-09T21:10:58.031 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.012+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:58.133 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.100+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:10:58.155 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.136+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:58.177 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.152+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:58.179 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.161+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:58.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.191+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:10:58.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.253+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:58.293 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.269+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:58.303 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:10:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.656', time=2019-12-09T23:07:29.554019+0000, chain.objs.size()=9 2019-12-09T21:10:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.29__multipart_key1.2~XXh44oV3l2SPzxYJKGDQ-IhLLTnfn0j.1 2019-12-09T21:10:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.29__shadow_key1.2~XXh44oV3l2SPzxYJKGDQ-IhLLTnfn0j.1_1 2019-12-09T21:10:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.29__multipart_key1.2~XXh44oV3l2SPzxYJKGDQ-IhLLTnfn0j.2 2019-12-09T21:10:58.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.29__shadow_key1.2~XXh44oV3l2SPzxYJKGDQ-IhLLTnfn0j.2_1 2019-12-09T21:10:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.29__multipart_key1.2~XXh44oV3l2SPzxYJKGDQ-IhLLTnfn0j.3 2019-12-09T21:10:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.29__shadow_key1.2~XXh44oV3l2SPzxYJKGDQ-IhLLTnfn0j.3_1 2019-12-09T21:10:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.29__multipart_key1.2~XXh44oV3l2SPzxYJKGDQ-IhLLTnfn0j.4 2019-12-09T21:10:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.29__shadow_key1.2~XXh44oV3l2SPzxYJKGDQ-IhLLTnfn0j.4_1 2019-12-09T21:10:58.305 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.283+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.29__multipart_key1.2~XXh44oV3l2SPzxYJKGDQ-IhLLTnfn0j.5 2019-12-09T21:10:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.576+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:58.721 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.703+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:10:58.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.755+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:58.796 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.778+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:58.805 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.786+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:58.857 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.839+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:10:58.908 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.890+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.910+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:58.941 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.923+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:58.978 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:58.959+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:10:59.066 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.047+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:59.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.068+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:59.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.079+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:59.142 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.122+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:10:59.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.173+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:59.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.191+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:59.218 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.200+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:59.254 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.236+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:10:59.293 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.273+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:59.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.290+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:59.323 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:10:59.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.438', time=2019-12-09T22:53:00.495842+0000, chain.objs.size()=9 2019-12-09T21:10:59.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~HejQhCUw5gBVagdoMurqP1WpZLQNaap.1 2019-12-09T21:10:59.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~HejQhCUw5gBVagdoMurqP1WpZLQNaap.1_1 2019-12-09T21:10:59.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~HejQhCUw5gBVagdoMurqP1WpZLQNaap.2 2019-12-09T21:10:59.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~HejQhCUw5gBVagdoMurqP1WpZLQNaap.2_1 2019-12-09T21:10:59.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~HejQhCUw5gBVagdoMurqP1WpZLQNaap.3 2019-12-09T21:10:59.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~HejQhCUw5gBVagdoMurqP1WpZLQNaap.3_1 2019-12-09T21:10:59.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~HejQhCUw5gBVagdoMurqP1WpZLQNaap.4 2019-12-09T21:10:59.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~HejQhCUw5gBVagdoMurqP1WpZLQNaap.4_1 2019-12-09T21:10:59.326 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.305+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~HejQhCUw5gBVagdoMurqP1WpZLQNaap.5 2019-12-09T21:10:59.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.648+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:10:59.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.748+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:10:59.806 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.787+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:10:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.807+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:10:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:10:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.556', time=2019-12-09T22:57:10.074280+0000, chain.objs.size()=9 2019-12-09T21:10:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file2.mpg.2~Er7tfKemjcdM3nO8gl0_QFPsfp8x9IY.1 2019-12-09T21:10:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__shadow_file2.mpg.2~Er7tfKemjcdM3nO8gl0_QFPsfp8x9IY.1_1 2019-12-09T21:10:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file2.mpg.2~Er7tfKemjcdM3nO8gl0_QFPsfp8x9IY.2 2019-12-09T21:10:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__shadow_file2.mpg.2~Er7tfKemjcdM3nO8gl0_QFPsfp8x9IY.2_1 2019-12-09T21:10:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file2.mpg.2~Er7tfKemjcdM3nO8gl0_QFPsfp8x9IY.3 2019-12-09T21:10:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__shadow_file2.mpg.2~Er7tfKemjcdM3nO8gl0_QFPsfp8x9IY.3_1 2019-12-09T21:10:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file2.mpg.2~Er7tfKemjcdM3nO8gl0_QFPsfp8x9IY.4 2019-12-09T21:10:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.845+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__shadow_file2.mpg.2~Er7tfKemjcdM3nO8gl0_QFPsfp8x9IY.4_1 2019-12-09T21:10:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.846+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.26__multipart_file2.mpg.2~Er7tfKemjcdM3nO8gl0_QFPsfp8x9IY.5 2019-12-09T21:10:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.846+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.605', time=2019-12-09T23:02:31.010343+0000, chain.objs.size()=21 2019-12-09T21:10:59.866 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.846+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.1 2019-12-09T21:10:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.846+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.1_1 2019-12-09T21:10:59.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:10:59.846+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.2 2019-12-09T21:11:00.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.016+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.2_1 2019-12-09T21:11:00.151 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.133+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.3 2019-12-09T21:11:00.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.140+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.3_1 2019-12-09T21:11:00.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.140+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.4 2019-12-09T21:11:00.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.140+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.4_1 2019-12-09T21:11:00.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.140+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.5 2019-12-09T21:11:00.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.140+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.5_1 2019-12-09T21:11:00.159 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.140+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.6 2019-12-09T21:11:00.159 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.140+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.6_1 2019-12-09T21:11:00.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.184+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.7 2019-12-09T21:11:00.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.184+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.7_1 2019-12-09T21:11:00.265 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.246+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.8 2019-12-09T21:11:00.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.398+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.8_1 2019-12-09T21:11:00.436 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.418+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.9 2019-12-09T21:11:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.418+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.9_1 2019-12-09T21:11:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.418+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.10 2019-12-09T21:11:00.437 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.418+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.10_1 2019-12-09T21:11:00.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.479+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~G0GhIQ8K4C0kjq95I7coKALbYQJ-CL4.11 2019-12-09T21:11:00.779 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.761+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:11:00.887 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.869+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:11:00.922 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.904+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:00.937 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.919+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:11:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.509', time=2019-12-09T22:56:09.728997+0000, chain.objs.size()=9 2019-12-09T21:11:00.957 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.24__multipart_key1.2~MzuoahyWIFRi6NBXoCfevtduEas1UUA.1 2019-12-09T21:11:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.24__shadow_key1.2~MzuoahyWIFRi6NBXoCfevtduEas1UUA.1_1 2019-12-09T21:11:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.24__multipart_key1.2~MzuoahyWIFRi6NBXoCfevtduEas1UUA.2 2019-12-09T21:11:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.24__shadow_key1.2~MzuoahyWIFRi6NBXoCfevtduEas1UUA.2_1 2019-12-09T21:11:00.958 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.24__multipart_key1.2~MzuoahyWIFRi6NBXoCfevtduEas1UUA.3 2019-12-09T21:11:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.24__shadow_key1.2~MzuoahyWIFRi6NBXoCfevtduEas1UUA.3_1 2019-12-09T21:11:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.24__multipart_key1.2~MzuoahyWIFRi6NBXoCfevtduEas1UUA.4 2019-12-09T21:11:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.24__shadow_key1.2~MzuoahyWIFRi6NBXoCfevtduEas1UUA.4_1 2019-12-09T21:11:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.24__multipart_key1.2~MzuoahyWIFRi6NBXoCfevtduEas1UUA.5 2019-12-09T21:11:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.617', time=2019-12-09T23:04:21.183412+0000, chain.objs.size()=19 2019-12-09T21:11:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.1 2019-12-09T21:11:00.959 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.1_1 2019-12-09T21:11:00.960 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:00.939+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.2 2019-12-09T21:11:01.208 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.189+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.2_1 2019-12-09T21:11:01.208 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.189+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.3 2019-12-09T21:11:01.208 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.189+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.3_1 2019-12-09T21:11:01.208 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.189+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.4 2019-12-09T21:11:01.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.189+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.4_1 2019-12-09T21:11:01.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.189+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.5 2019-12-09T21:11:01.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.189+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.5_1 2019-12-09T21:11:01.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.300+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.6 2019-12-09T21:11:01.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.300+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.6_1 2019-12-09T21:11:01.481 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.463+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.7 2019-12-09T21:11:01.481 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.463+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.7_1 2019-12-09T21:11:01.482 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.463+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.8 2019-12-09T21:11:01.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.600+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.8_1 2019-12-09T21:11:01.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.600+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.9 2019-12-09T21:11:01.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.600+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.9_1 2019-12-09T21:11:01.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.600+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~7cDKEtXbDsGOLSIILhXsh-mSgxZGe2Y.10 2019-12-09T21:11:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.855+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:11:01.966 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.948+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:11:02.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:01.988+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:02.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.003+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:02.030 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.011+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:02.077 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.059+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:11:02.156 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.138+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:02.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.176+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:02.210 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.191+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:02.263 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.244+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:11:02.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.301+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:02.335 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.317+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:02.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.324+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:02.376 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.357+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:11:02.453 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.435+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:02.475 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.457+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:11:02.499 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh', time=2019-12-09T22:49:36.835967+0000, chain.objs.size()=9 2019-12-09T21:11:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.13__multipart_key1.2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh.1 2019-12-09T21:11:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.13__shadow_key1.2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh.1_1 2019-12-09T21:11:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.13__multipart_key1.2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh.2 2019-12-09T21:11:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.13__shadow_key1.2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh.2_1 2019-12-09T21:11:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.13__multipart_key1.2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh.3 2019-12-09T21:11:02.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.13__shadow_key1.2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh.3_1 2019-12-09T21:11:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.13__multipart_key1.2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh.4 2019-12-09T21:11:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.13__shadow_key1.2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh.4_1 2019-12-09T21:11:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.13__multipart_key1.2~6GnWcZYP2tFw7lZ20xLiEKW_al9femh.5 2019-12-09T21:11:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.446', time=2019-12-09T22:53:31.642402+0000, chain.objs.size()=9 2019-12-09T21:11:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~RNscbbU482_8CL-WBd4CMNgoTCVB4Td.1 2019-12-09T21:11:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~RNscbbU482_8CL-WBd4CMNgoTCVB4Td.1_1 2019-12-09T21:11:02.501 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.478+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~RNscbbU482_8CL-WBd4CMNgoTCVB4Td.2 2019-12-09T21:11:02.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.713+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~RNscbbU482_8CL-WBd4CMNgoTCVB4Td.2_1 2019-12-09T21:11:02.802 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.783+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~RNscbbU482_8CL-WBd4CMNgoTCVB4Td.3 2019-12-09T21:11:02.802 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.784+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~RNscbbU482_8CL-WBd4CMNgoTCVB4Td.3_1 2019-12-09T21:11:02.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.790+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~RNscbbU482_8CL-WBd4CMNgoTCVB4Td.4 2019-12-09T21:11:02.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.791+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~RNscbbU482_8CL-WBd4CMNgoTCVB4Td.4_1 2019-12-09T21:11:02.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:02.791+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~RNscbbU482_8CL-WBd4CMNgoTCVB4Td.5 2019-12-09T21:11:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.122+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:11:03.229 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.210+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:11:03.301 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.283+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:03.332 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.312+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:11:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.454', time=2019-12-09T22:53:59.436003+0000, chain.objs.size()=9 2019-12-09T21:11:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~cNrBE5rTEJXIOIeOJTv-s_A2NzgL7am.1 2019-12-09T21:11:03.357 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~cNrBE5rTEJXIOIeOJTv-s_A2NzgL7am.1_1 2019-12-09T21:11:03.358 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~cNrBE5rTEJXIOIeOJTv-s_A2NzgL7am.2 2019-12-09T21:11:03.358 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~cNrBE5rTEJXIOIeOJTv-s_A2NzgL7am.2_1 2019-12-09T21:11:03.358 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~cNrBE5rTEJXIOIeOJTv-s_A2NzgL7am.3 2019-12-09T21:11:03.358 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~cNrBE5rTEJXIOIeOJTv-s_A2NzgL7am.3_1 2019-12-09T21:11:03.358 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~cNrBE5rTEJXIOIeOJTv-s_A2NzgL7am.4 2019-12-09T21:11:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~cNrBE5rTEJXIOIeOJTv-s_A2NzgL7am.4_1 2019-12-09T21:11:03.359 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.338+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~cNrBE5rTEJXIOIeOJTv-s_A2NzgL7am.5 2019-12-09T21:11:03.715 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.693+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:11:03.784 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.765+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:11:03.820 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.800+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:03.837 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.815+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:03.851 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:11:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.466', time=2019-12-09T22:55:01.556382+0000, chain.objs.size()=7 2019-12-09T21:11:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~BnNOKZTAQB1YmGAScQO2DSP5pHhFfai.1 2019-12-09T21:11:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~BnNOKZTAQB1YmGAScQO2DSP5pHhFfai.1_1 2019-12-09T21:11:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~BnNOKZTAQB1YmGAScQO2DSP5pHhFfai.1_2 2019-12-09T21:11:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~BnNOKZTAQB1YmGAScQO2DSP5pHhFfai.2 2019-12-09T21:11:03.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~BnNOKZTAQB1YmGAScQO2DSP5pHhFfai.2_1 2019-12-09T21:11:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__shadow_key1.2~BnNOKZTAQB1YmGAScQO2DSP5pHhFfai.2_2 2019-12-09T21:11:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.20__multipart_key1.2~BnNOKZTAQB1YmGAScQO2DSP5pHhFfai.3 2019-12-09T21:11:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.625', time=2019-12-09T23:05:23.321489+0000, chain.objs.size()=16 2019-12-09T21:11:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.1 2019-12-09T21:11:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.1_1 2019-12-09T21:11:03.853 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.1_2 2019-12-09T21:11:03.854 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.2 2019-12-09T21:11:03.854 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:03.831+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.2_1 2019-12-09T21:11:04.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.2_2 2019-12-09T21:11:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.3 2019-12-09T21:11:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.3_1 2019-12-09T21:11:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.3_2 2019-12-09T21:11:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.4 2019-12-09T21:11:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.4_1 2019-12-09T21:11:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.4_2 2019-12-09T21:11:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.5 2019-12-09T21:11:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.5_1 2019-12-09T21:11:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.5_2 2019-12-09T21:11:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.207+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~w9WGdhtndswK45hrj-Md5cT86kSx02n.6 2019-12-09T21:11:04.702 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.683+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:11:04.949 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:04.930+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:11:05.026 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.008+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:05.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.034+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:05.072 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:11:05.072 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.417', time=2019-12-09T22:51:07.609781+0000, chain.objs.size()=9 2019-12-09T21:11:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.1 2019-12-09T21:11:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__shadow_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.1_1 2019-12-09T21:11:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.2 2019-12-09T21:11:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__shadow_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.2_1 2019-12-09T21:11:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.3 2019-12-09T21:11:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__shadow_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.3_1 2019-12-09T21:11:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.4 2019-12-09T21:11:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__shadow_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.4_1 2019-12-09T21:11:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.054+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.5 2019-12-09T21:11:05.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.306+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:11:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.415+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:11:05.475 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.457+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:05.492 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.474+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.486+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:11:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.486+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.430', time=2019-12-09T22:55:10.850586+0000, chain.objs.size()=5 2019-12-09T21:11:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.486+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.19__shadow_.gND7uQj52NAv8gjMV718qNMIAZjJMg9_1 2019-12-09T21:11:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.486+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.19__shadow_.gND7uQj52NAv8gjMV718qNMIAZjJMg9_2 2019-12-09T21:11:05.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.486+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.19__shadow_.gND7uQj52NAv8gjMV718qNMIAZjJMg9_3 2019-12-09T21:11:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.486+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.19__shadow_.gND7uQj52NAv8gjMV718qNMIAZjJMg9_4 2019-12-09T21:11:05.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.486+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.19__shadow_.gND7uQj52NAv8gjMV718qNMIAZjJMg9_5 2019-12-09T21:11:05.717 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.699+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:11:05.884 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.865+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:11:05.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.908+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:05.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.926+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:05.955 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.937+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:05.988 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:05.969+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:11:06.039 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.021+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:06.056 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.038+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:06.065 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.047+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:06.111 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.093+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:11:06.173 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.155+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:06.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.176+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:06.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.184+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:06.273 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.252+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:11:06.310 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.289+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:06.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.305+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:06.348 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.326+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:11:06.348 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.326+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.415', time=2019-12-09T22:51:12.984445+0000, chain.objs.size()=9 2019-12-09T21:11:06.348 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.326+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.1 2019-12-09T21:11:06.348 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__shadow_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.1_1 2019-12-09T21:11:06.349 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.2 2019-12-09T21:11:06.349 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__shadow_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.2_1 2019-12-09T21:11:06.349 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.3 2019-12-09T21:11:06.349 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__shadow_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.3_1 2019-12-09T21:11:06.349 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.4 2019-12-09T21:11:06.349 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__shadow_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.4_1 2019-12-09T21:11:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.17__multipart_key1.2~p1a7pgXjK3HlSbUMaTacNbCE7h1Vawk.5 2019-12-09T21:11:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.633', time=2019-12-09T23:06:20.147450+0000, chain.objs.size()=16 2019-12-09T21:11:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.1 2019-12-09T21:11:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.1_1 2019-12-09T21:11:06.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.327+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.1_2 2019-12-09T21:11:06.566 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.549+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.2 2019-12-09T21:11:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.2_1 2019-12-09T21:11:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.2_2 2019-12-09T21:11:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.3 2019-12-09T21:11:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.3_1 2019-12-09T21:11:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.3_2 2019-12-09T21:11:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.4 2019-12-09T21:11:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.4_1 2019-12-09T21:11:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.4_2 2019-12-09T21:11:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.5 2019-12-09T21:11:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.5_1 2019-12-09T21:11:06.787 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:06.766+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.5_2 2019-12-09T21:11:07.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.007+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EkCwN8pvWeaemM40PBz1azFRQH0o-Oy.6 2019-12-09T21:11:07.162 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.145+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:11:07.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.301+0000 7fbb96767ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:11:07.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.347+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:07.381 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.363+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:07.409 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:11:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.592', time=2019-12-09T23:01:14.377914+0000, chain.objs.size()=21 2019-12-09T21:11:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.1 2019-12-09T21:11:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.1_1 2019-12-09T21:11:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.2 2019-12-09T21:11:07.411 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.2_1 2019-12-09T21:11:07.411 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.3 2019-12-09T21:11:07.411 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.3_1 2019-12-09T21:11:07.411 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.4 2019-12-09T21:11:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.4_1 2019-12-09T21:11:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.5 2019-12-09T21:11:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.5_1 2019-12-09T21:11:07.412 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.6 2019-12-09T21:11:07.413 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.388+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.6_1 2019-12-09T21:11:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.663+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.7 2019-12-09T21:11:07.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.663+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.7_1 2019-12-09T21:11:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.8 2019-12-09T21:11:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.8_1 2019-12-09T21:11:07.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.9 2019-12-09T21:11:07.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.9_1 2019-12-09T21:11:07.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.10 2019-12-09T21:11:07.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.10_1 2019-12-09T21:11:07.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~EbWsL1xPuMC1h_qQqHTGrNcfTmhsDPG.11 2019-12-09T21:11:07.800 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.641', time=2019-12-09T23:06:22.197242+0000, chain.objs.size()=16 2019-12-09T21:11:07.800 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.1 2019-12-09T21:11:07.800 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.780+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.1_1 2019-12-09T21:11:07.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.1_2 2019-12-09T21:11:07.990 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:07.973+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.2 2019-12-09T21:11:08.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.170+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.2_1 2019-12-09T21:11:08.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.2_2 2019-12-09T21:11:08.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.3 2019-12-09T21:11:08.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.3_1 2019-12-09T21:11:08.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.3_2 2019-12-09T21:11:08.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.4 2019-12-09T21:11:08.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.4_1 2019-12-09T21:11:08.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.4_2 2019-12-09T21:11:08.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.5 2019-12-09T21:11:08.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.179+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.5_1 2019-12-09T21:11:08.260 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.241+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__shadow_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.5_2 2019-12-09T21:11:08.402 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.384+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.28__multipart_key1.2~fx8Hmi-GO7oZN1QhMiKaWwPxlLi8OOd.6 2019-12-09T21:11:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.572+0000 7fbb96767ec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:11:08.763 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.746+0000 7fbb96767ec0 20 remove_watcher() i=0 2019-12-09T21:11:08.764 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.746+0000 7fbb96767ec0 2 removed watcher, disabling cache 2019-12-09T21:11:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.786+0000 7fbb96767ec0 20 remove_watcher() i=1 2019-12-09T21:11:08.850 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.832+0000 7fbb96767ec0 20 remove_watcher() i=2 2019-12-09T21:11:08.897 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.878+0000 7fbb96767ec0 20 remove_watcher() i=3 2019-12-09T21:11:08.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.927+0000 7fbb96767ec0 20 remove_watcher() i=4 2019-12-09T21:11:09.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:08.988+0000 7fbb96767ec0 20 remove_watcher() i=5 2019-12-09T21:11:09.041 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.024+0000 7fbb96767ec0 20 remove_watcher() i=6 2019-12-09T21:11:09.091 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.073+0000 7fbb96767ec0 20 remove_watcher() i=7 2019-12-09T21:11:09.096 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:11:09.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.192+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.232 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.215+0000 7fc572f24ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:09.233 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.215+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.223+0000 7fc572f24ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:09.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.223+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.227+0000 7fc572f24ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:09.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.227+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.255 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.238+0000 7fc572f24ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:09.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.374+0000 7fc572f24ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:11:09.392 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.374+0000 7fc572f24ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:11:09.392 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.374+0000 7fc572f24ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:11:09.392 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.374+0000 7fc572f24ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:11:09.392 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.374+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.384+0000 7fc572f24ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.384+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.393+0000 7fc572f24ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:11:09.411 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.393+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.406+0000 7fc572f24ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.406+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.437 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.416+0000 7fc572f24ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:11:09.438 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.416+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.432+0000 7fc572f24ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.432+0000 7fc572f24ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:11:09.450 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.432+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.437+0000 7fc572f24ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:09.456 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.438+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.462 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.445+0000 7fc572f24ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:09.463 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.445+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.472 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.454+0000 7fc572f24ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:11:09.472 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.454+0000 7fc572f24ec0 20 zonegroup default 2019-12-09T21:11:09.473 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.454+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.483 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.465+0000 7fc572f24ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:09.483 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.465+0000 7fc572f24ec0 10 Cannot find current period zone using local zone 2019-12-09T21:11:09.484 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.465+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.473+0000 7fc572f24ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:09.490 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.473+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.485+0000 7fc572f24ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:09.502 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.485+0000 7fc572f24ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.494+0000 7fc572f24ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:11:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.494+0000 7fc572f24ec0 20 zone default found 2019-12-09T21:11:09.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.494+0000 7fc572f24ec0 20 started sync module instance, tier type = 2019-12-09T21:11:09.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.494+0000 7fc572f24ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:11:09.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.861+0000 7fc572f24ec0 20 add_watcher() i=0 2019-12-09T21:11:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.861+0000 7fc572f24ec0 20 add_watcher() i=1 2019-12-09T21:11:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.861+0000 7fc572f24ec0 20 add_watcher() i=2 2019-12-09T21:11:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.861+0000 7fc572f24ec0 20 add_watcher() i=3 2019-12-09T21:11:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.861+0000 7fc572f24ec0 20 add_watcher() i=4 2019-12-09T21:11:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.861+0000 7fc572f24ec0 20 add_watcher() i=5 2019-12-09T21:11:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.861+0000 7fc572f24ec0 20 add_watcher() i=6 2019-12-09T21:11:09.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.909+0000 7fc572f24ec0 20 add_watcher() i=7 2019-12-09T21:11:09.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.909+0000 7fc572f24ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:11:09.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.909+0000 7fc530fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:11:09.930 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.913+0000 7fc56014e700 0 -- 172.21.15.67:0/3459679750 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x559dc3082220 msgr2=0x559dc30521d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send injecting socket failure 2019-12-09T21:11:09.986 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:09.970+0000 7fc55f94d700 0 -- 172.21.15.67:0/2448179785 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x559dc3096510 msgr2=0x559dc3098930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:11:11.421 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.404+0000 7fc4a2ffd700 20 reqs_thread_entry: start 2019-12-09T21:11:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.404+0000 7fc4a0ff9700 20 reqs_thread_entry: start 2019-12-09T21:11:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.404+0000 7fc572f24ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:11:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.405+0000 7fc496ffd700 20 reqs_thread_entry: start 2019-12-09T21:11:11.422 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.405+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:11:11.468 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.450+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:11.485 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.469+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:11.495 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.478+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:11.522 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.506+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:11:11.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.568+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:11.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.587+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.596+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:11.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.626+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:11:11.710 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.693+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:11.727 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.709+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:11.735 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.718+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:11.792 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.775+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:11:11.855 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.837+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:11.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.855+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:11.883 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.866+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:11.930 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.913+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:11:12.016 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:11.993+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.044 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.027+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.037+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.084+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:11:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.133+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.167 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.150+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:12.178 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.161+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.196+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:11:12.262 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.245+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.285 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.267+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:12.296 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.276+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.327 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.310+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:11:12.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.367+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.388+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:12.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.397+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.468 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.451+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:11:12.539 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.522+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.564 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.547+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:12.580 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.561+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.601+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:11:12.677 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.660+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.714 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.697+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:12.728 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.711+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.763 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.746+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:11:12.821 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.804+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.846 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.829+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:12.861 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.844+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.909 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.892+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:11:12.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.954+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:12.991 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.974+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:13.002 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:12.985+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.038 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.021+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:11:13.080 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.063+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.078+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:13.118 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.087+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.120+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:11:13.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.204+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.239+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:13.303 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.260+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.325+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:11:13.440 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.416+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.471 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.450+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:13.485 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.467+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.559 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.542+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:11:13.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.607+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.631+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:13.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.639+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.699 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.682+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:11:13.762 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.742+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.788 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.771+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:13.804 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.787+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.858 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.841+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:11:13.916 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.899+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.948 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.923+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:13.953 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.932+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:13.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:13.975+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:11:14.057 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.036+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.072 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.056+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:14.083 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.065+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.121+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:11:14.236 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.219+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.263 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.246+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:14.280 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.262+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.307+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:11:14.378 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.360+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.395 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.377+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:14.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.386+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.452 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.435+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:11:14.539 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.522+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.561 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.544+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:14.573 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.552+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.605+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:11:14.685 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.666+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.694+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.704+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.773 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.753+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:11:14.833 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.814+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.865 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.848+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:14.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.861+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:14.928 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.911+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:11:14.997 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:14.980+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.000+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:15.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.009+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.055+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:11:15.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.108+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.124+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:15.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.131+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.174+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:11:15.253 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.236+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.271 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.254+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:15.283 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.263+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.296+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:11:15.363 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.346+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.382 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.364+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:15.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.373+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.435 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.418+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:11:15.486 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.469+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.502 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.485+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:15.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.495+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.550 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.533+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:11:15.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.611+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.638+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:15.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.650+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.714 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.697+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:11:15.764 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.745+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.770+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:15.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.782+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.824+0000 7fc572f24ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:11:15.894 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.877+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.911 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.894+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:15.919 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.902+0000 7fc572f24ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.976+0000 7fc572f24ec0 20 remove_watcher() i=0 2019-12-09T21:11:15.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:15.976+0000 7fc572f24ec0 2 removed watcher, disabling cache 2019-12-09T21:11:16.043 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.026+0000 7fc572f24ec0 20 remove_watcher() i=1 2019-12-09T21:11:16.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.065+0000 7fc572f24ec0 20 remove_watcher() i=2 2019-12-09T21:11:16.147 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.130+0000 7fc572f24ec0 20 remove_watcher() i=3 2019-12-09T21:11:16.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.177+0000 7fc572f24ec0 20 remove_watcher() i=4 2019-12-09T21:11:16.241 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.224+0000 7fc572f24ec0 20 remove_watcher() i=5 2019-12-09T21:11:16.303 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.286+0000 7fc572f24ec0 20 remove_watcher() i=6 2019-12-09T21:11:16.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.324+0000 7fc572f24ec0 20 remove_watcher() i=7 2019-12-09T21:11:16.353 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:11:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.522+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.574 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.557+0000 7f22af78eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:16.574 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.557+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.567+0000 7f22af78eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:16.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.567+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.590 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.574+0000 7f22af78eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:16.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.574+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.582+0000 7f22af78eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:16.794 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.777+0000 7f22af78eec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:11:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.777+0000 7f22af78eec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:11:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.777+0000 7f22af78eec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:11:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.777+0000 7f22af78eec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:11:16.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.777+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.807 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.790+0000 7f22af78eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:16.807 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.790+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.820 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.803+0000 7f22af78eec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:11:16.820 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.803+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.814+0000 7f22af78eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:16.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.814+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.824+0000 7f22af78eec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:11:16.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.825+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.836+0000 7f22af78eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:16.853 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.836+0000 7f22af78eec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:11:16.854 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.836+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.862 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.845+0000 7f22af78eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:16.862 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.845+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.855+0000 7f22af78eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:16.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.855+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.882 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.865+0000 7f22af78eec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:11:16.882 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.865+0000 7f22af78eec0 20 zonegroup default 2019-12-09T21:11:16.882 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.865+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.888 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.871+0000 7f22af78eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:16.888 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.871+0000 7f22af78eec0 10 Cannot find current period zone using local zone 2019-12-09T21:11:16.888 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.871+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.893 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.877+0000 7f22af78eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:16.893 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.877+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.902 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.886+0000 7f22af78eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:16.902 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.886+0000 7f22af78eec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:16.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.896+0000 7f22af78eec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:11:16.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.896+0000 7f22af78eec0 20 zone default found 2019-12-09T21:11:16.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.896+0000 7f22af78eec0 20 started sync module instance, tier type = 2019-12-09T21:11:16.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:16.896+0000 7f22af78eec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:11:17.255 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.238+0000 7f22af78eec0 20 add_watcher() i=0 2019-12-09T21:11:17.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.238+0000 7f22af78eec0 20 add_watcher() i=1 2019-12-09T21:11:17.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.238+0000 7f22af78eec0 20 add_watcher() i=2 2019-12-09T21:11:17.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.238+0000 7f22af78eec0 20 add_watcher() i=3 2019-12-09T21:11:17.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.238+0000 7f22af78eec0 20 add_watcher() i=4 2019-12-09T21:11:17.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.238+0000 7f22af78eec0 20 add_watcher() i=5 2019-12-09T21:11:17.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.238+0000 7f22af78eec0 20 add_watcher() i=6 2019-12-09T21:11:17.292 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.275+0000 7f22af78eec0 20 add_watcher() i=7 2019-12-09T21:11:17.292 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.275+0000 7f22af78eec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:11:17.292 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.275+0000 7f226d7e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:11:17.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.291+0000 7f229c1b7700 0 -- 172.21.15.67:0/2461769638 >> v2:172.21.15.67:3300/0 conn(0x558604a281d0 msgr2=0x558604a1fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:11:17.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:17.397+0000 7f229d1b9700 0 -- 172.21.15.67:0/2461769638 >> [v2:172.21.15.67:6802/10705,v1:172.21.15.67:6803/10705] conn(0x558604a665e0 msgr2=0x558604a68a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:11:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:18.803+0000 7f21df7fe700 20 reqs_thread_entry: start 2019-12-09T21:11:18.821 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:18.803+0000 7f21dd7fa700 20 reqs_thread_entry: start 2019-12-09T21:11:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:18.803+0000 7f22af78eec0 20 init_complete bucket index max shards: 0 2019-12-09T21:11:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:18.804+0000 7f21cb7fe700 20 reqs_thread_entry: start 2019-12-09T21:11:18.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:18.804+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:11:18.903 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:18.886+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:18.933 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:18.916+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:18.949 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:18.932+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.004 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:18.987+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:11:19.090 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.070+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.106+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:19.139 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.122+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.189 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.172+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:11:19.284 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.266+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.292+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:19.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.307+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.371+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:11:19.473 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.456+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.527 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.510+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:19.560 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.543+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.589+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:11:19.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.649+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.686 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.669+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:19.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.681+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.719+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:11:19.791 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.775+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.812 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.795+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:19.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.806+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.853+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:11:19.934 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.916+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:19.956 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.939+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:19.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.954+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.008 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:19.991+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:11:20.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.079+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.125 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.108+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.125+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.180+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:11:20.255 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.238+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.259+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:20.317 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.269+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.332 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.315+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:11:20.399 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.381+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.400+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:20.426 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.410+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.470 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.453+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:11:20.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.495+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.528 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.511+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:20.536 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.519+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.565 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.548+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:11:20.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.590+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.606+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:20.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.614+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.650+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:11:20.718 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.701+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.719+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:20.751 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.731+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.782 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.765+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:11:20.857 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.840+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.893 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.876+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:20.912 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.892+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:20.957 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.937+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:11:21.007 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:20.989+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.035 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.008+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:21.035 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.018+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.089 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.059+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:11:21.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.132+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.168 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.151+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.164+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.217+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:11:21.323 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.305+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.328+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.349+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.391+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:11:21.453 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.435+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.472 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.453+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:21.482 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.465+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.526 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.509+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:11:21.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.593+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.609+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:21.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.617+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.658+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:11:21.724 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.706+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.721+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:21.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.730+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.781+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:11:21.862 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.845+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.861+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:21.886 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.869+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.939+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:11:22.001 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.983+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.015 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:21.998+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:22.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.007+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.062 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.045+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:11:22.108 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.087+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.122 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.105+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:22.138 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.120+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.169 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.152+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:11:22.229 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.212+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.230+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:22.257 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.241+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.316 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.293+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:11:22.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.374+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.421 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.404+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:22.452 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.429+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.499 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.482+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:11:22.560 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.532+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.565 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.548+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:22.576 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.559+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.604+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:11:22.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.688+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.735 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.718+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:22.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.731+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.784+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:11:22.874 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.857+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.896 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.879+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:22.908 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.891+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:22.949 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.931+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:11:22.997 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:22.979+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:23.018 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.000+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:23.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.012+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:23.068 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.051+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:11:23.332 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.315+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:23.371 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.353+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:23.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.368+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:23.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.428+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:11:23.513 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.496+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:23.541 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.523+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:23.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.535+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:23.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.578+0000 7f22af78eec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:11:23.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.638+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:23.678 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.662+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:23.691 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.675+0000 7f22af78eec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.772+0000 7f22af78eec0 20 remove_watcher() i=0 2019-12-09T21:11:23.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.772+0000 7f22af78eec0 2 removed watcher, disabling cache 2019-12-09T21:11:23.848 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.831+0000 7f22af78eec0 20 remove_watcher() i=1 2019-12-09T21:11:23.908 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.891+0000 7f22af78eec0 20 remove_watcher() i=2 2019-12-09T21:11:23.996 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:23.979+0000 7f22af78eec0 20 remove_watcher() i=3 2019-12-09T21:11:24.074 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.041+0000 7f22af78eec0 20 remove_watcher() i=4 2019-12-09T21:11:24.107 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.089+0000 7f22af78eec0 20 remove_watcher() i=5 2019-12-09T21:11:24.160 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.143+0000 7f22af78eec0 20 remove_watcher() i=6 2019-12-09T21:11:24.215 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.198+0000 7f22af78eec0 20 remove_watcher() i=7 2019-12-09T21:11:24.227 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:11:24.386 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.368+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.399+0000 7f8ad55d3ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:24.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.399+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.434 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.418+0000 7f8ad55d3ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:24.435 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.418+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.423+0000 7f8ad55d3ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:24.446 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.423+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.451 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.434+0000 7f8ad55d3ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.633+0000 7f8ad55d3ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:11:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.633+0000 7f8ad55d3ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:11:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.633+0000 7f8ad55d3ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:11:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.633+0000 7f8ad55d3ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:11:24.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.633+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.646+0000 7f8ad55d3ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.646+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.678 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.660+0000 7f8ad55d3ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:11:24.678 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.660+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.671+0000 7f8ad55d3ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:24.688 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.671+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.702 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.684+0000 7f8ad55d3ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:11:24.702 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.684+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.711 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.693+0000 7f8ad55d3ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:24.711 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.693+0000 7f8ad55d3ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:11:24.711 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.693+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.717 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.700+0000 7f8ad55d3ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:24.718 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.700+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.715+0000 7f8ad55d3ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:24.732 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.715+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.728+0000 7f8ad55d3ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:11:24.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.728+0000 7f8ad55d3ec0 20 zonegroup default 2019-12-09T21:11:24.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.728+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.751 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.735+0000 7f8ad55d3ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:24.752 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.735+0000 7f8ad55d3ec0 10 Cannot find current period zone using local zone 2019-12-09T21:11:24.752 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.735+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.743+0000 7f8ad55d3ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:11:24.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.743+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.768 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.751+0000 7f8ad55d3ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:11:24.768 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.751+0000 7f8ad55d3ec0 20 rados->read ofs=0 len=0 2019-12-09T21:11:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.763+0000 7f8ad55d3ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:11:24.780 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.763+0000 7f8ad55d3ec0 20 zone default found 2019-12-09T21:11:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.763+0000 7f8ad55d3ec0 20 started sync module instance, tier type = 2019-12-09T21:11:24.781 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:24.763+0000 7f8ad55d3ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:11:25.156 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.139+0000 7f8ad55d3ec0 20 add_watcher() i=0 2019-12-09T21:11:25.157 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.139+0000 7f8ad55d3ec0 20 add_watcher() i=1 2019-12-09T21:11:25.157 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.139+0000 7f8ad55d3ec0 20 add_watcher() i=2 2019-12-09T21:11:25.157 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.139+0000 7f8ad55d3ec0 20 add_watcher() i=3 2019-12-09T21:11:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.139+0000 7f8ad55d3ec0 20 add_watcher() i=4 2019-12-09T21:11:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.139+0000 7f8ad55d3ec0 20 add_watcher() i=5 2019-12-09T21:11:25.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.139+0000 7f8ad55d3ec0 20 add_watcher() i=6 2019-12-09T21:11:25.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.183+0000 7f8ad55d3ec0 20 add_watcher() i=7 2019-12-09T21:11:25.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.183+0000 7f8ad55d3ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:11:25.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.183+0000 7f8a92fdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:11:25.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.186+0000 7f8ac2ffe700 0 -- 172.21.15.67:0/3714197746 >> v1:172.21.15.81:6789/0 conn(0x55989fe7a300 legacy=0x55989fe5d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:11:25.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:25.223+0000 7f8ac27fd700 0 -- 172.21.15.67:0/2112955716 >> [v2:172.21.15.67:6800/9311,v1:172.21.15.67:6801/9311] conn(0x7f8a04022470 msgr2=0x7f8a04024980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:11:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.628+0000 7f8a017fa700 20 reqs_thread_entry: start 2019-12-09T21:11:26.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.629+0000 7f89faffd700 20 reqs_thread_entry: start 2019-12-09T21:11:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.629+0000 7f8ad55d3ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:11:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.629+0000 7f89f97fa700 20 reqs_thread_entry: start 2019-12-09T21:11:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.629+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:11:26.716 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.699+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:26.737 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.720+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:26.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.729+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:26.780 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.763+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:11:26.840 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.823+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:26.855 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.838+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:26.864 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.848+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:26.917 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.900+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:11:26.975 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.958+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:26.992 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.975+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:27.003 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:26.986+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.049 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.018+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:11:27.078 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.061+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.093 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.077+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:27.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.086+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.217+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:11:27.294 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.277+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.310 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.293+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:27.320 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.302+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.358 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.341+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:11:27.425 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.408+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.454 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.435+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:27.468 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.451+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.540 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.523+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:11:27.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.633+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.670+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:27.700 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.684+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.735 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.719+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:11:27.815 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.798+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.849 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.832+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:27.864 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.847+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:27.941 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.918+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:11:28.002 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:27.979+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.018 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.001+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:28.036 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.014+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.070 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.053+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:11:28.275 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.258+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.301 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.284+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:28.312 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.296+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.339+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:11:28.448 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.431+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.489 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.467+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.488+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.565 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.548+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:11:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.634+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.677 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.660+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:28.707 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.679+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.762 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.745+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:11:28.835 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.818+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.858 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.841+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:28.900 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.882+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:28.947 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:28.930+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:11:29.038 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.021+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.077 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.058+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:29.096 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.079+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.156+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:11:29.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.223+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.269 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.251+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:29.281 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.264+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.335+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:11:29.467 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.449+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.496 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.479+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:29.510 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.493+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.550+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:11:29.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.637+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.665+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.683+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.729+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:11:29.808 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.791+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.828 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.811+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:29.849 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.832+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.904 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.882+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:11:29.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.948+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:29.993 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.976+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:30.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:29.989+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.036+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:11:30.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.084+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.119 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.102+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:30.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.112+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.146+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:11:30.234 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.209+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.251 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.234+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:30.265 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.248+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.290+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:11:30.357 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.340+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.357+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:30.383 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.367+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.434 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.418+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:11:30.518 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.502+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.544 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.527+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.539+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.581+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:11:30.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.624+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.643+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:30.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.654+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.719 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.702+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:11:30.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.756+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.780+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:30.806 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.789+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.824+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:11:30.883 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.865+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.904 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.887+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:30.915 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.898+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:30.956 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:30.939+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:11:31.031 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.014+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.056 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.038+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:31.069 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.053+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.120 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.103+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:11:31.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.159+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.182+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:31.211 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.194+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.262 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.245+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:11:31.311 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.295+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.329 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.313+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:31.339 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.322+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.379 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.363+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:11:31.435 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.418+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.463 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.447+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.459+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.513 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.497+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:11:31.574 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.557+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.576+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:31.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.589+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.616+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:11:31.683 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.667+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.699 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.683+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:11:31.708 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.691+0000 7f8ad55d3ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:11:31.831 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.815+0000 7f8ad55d3ec0 20 remove_watcher() i=0 2019-12-09T21:11:31.832 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.815+0000 7f8ad55d3ec0 2 removed watcher, disabling cache 2019-12-09T21:11:31.867 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.850+0000 7f8ad55d3ec0 20 remove_watcher() i=1 2019-12-09T21:11:31.900 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.884+0000 7f8ad55d3ec0 20 remove_watcher() i=2 2019-12-09T21:11:31.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.948+0000 7f8ad55d3ec0 20 remove_watcher() i=3 2019-12-09T21:11:32.003 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:31.986+0000 7f8ad55d3ec0 20 remove_watcher() i=4 2019-12-09T21:11:32.042 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:32.026+0000 7f8ad55d3ec0 20 remove_watcher() i=5 2019-12-09T21:11:32.083 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:32.064+0000 7f8ad55d3ec0 20 remove_watcher() i=6 2019-12-09T21:11:32.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:11:32.112+0000 7f8ad55d3ec0 20 remove_watcher() i=7 2019-12-09T21:11:32.141 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle-4.7/bin/gradle clean test --rerun-tasks --no-build-cache --tests BucketTest 2019-12-09T21:11:32.985 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :clean 2019-12-09T21:11:33.584 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :compileJava 2019-12-09T21:11:33.584 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :processResources 2019-12-09T21:11:33.584 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :classes 2019-12-09T21:11:33.984 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:33.984 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :compileTestJava 2019-12-09T21:11:33.984 INFO:teuthology.orchestra.run.smithi067.stdout:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2019-12-09T21:11:33.984 INFO:teuthology.orchestra.run.smithi067.stdout:Note: Recompile with -Xlint:deprecation for details. 2019-12-09T21:11:33.985 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:33.985 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :processTestResources NO-SOURCE 2019-12-09T21:11:33.985 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :testClasses 2019-12-09T21:11:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:36.183 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :test 2019-12-09T21:11:36.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:36.184 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectEmpty STARTED 2019-12-09T21:11:37.983 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:37.984 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectEmpty PASSED 2019-12-09T21:11:44.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:44.183 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectMismatch STARTED 2019-12-09T21:11:46.083 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:46.083 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectMismatch PASSED 2019-12-09T21:11:51.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:51.183 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectUnreadable STARTED 2019-12-09T21:11:53.284 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:53.284 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketCreateBadExpectUnreadable PASSED 2019-12-09T21:11:59.582 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:11:59.583 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketCreateReadDelete STARTED 2019-12-09T21:12:06.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:06.085 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketCreateReadDelete PASSED 2019-12-09T21:12:07.183 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:07.183 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNonEmpty STARTED 2019-12-09T21:12:11.584 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:11.584 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNonEmpty PASSED 2019-12-09T21:12:17.882 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:17.882 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNotExist STARTED 2019-12-09T21:12:18.682 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:18.683 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketDeleteNotExist PASSED 2019-12-09T21:12:19.682 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:19.683 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketListDistinct STARTED 2019-12-09T21:12:25.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:25.783 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketListDistinct PASSED 2019-12-09T21:12:37.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:37.783 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketListEmpty STARTED 2019-12-09T21:12:41.582 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:41.582 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketListEmpty PASSED 2019-12-09T21:12:46.682 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:46.682 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketNotExist STARTED 2019-12-09T21:12:46.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:46.782 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Bucket tests > BucketTest.testBucketNotExist PASSED 2019-12-09T21:12:47.354 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:47.354 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------- 2019-12-09T21:12:47.354 INFO:teuthology.orchestra.run.smithi067.stdout:| Results: SUCCESS (9 tests, 9 successes, 0 failures, 0 skipped) | 2019-12-09T21:12:47.354 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------- 2019-12-09T21:12:47.355 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:12:47.355 INFO:teuthology.orchestra.run.smithi067.stdout:BUILD SUCCESSFUL in 1m 15s 2019-12-09T21:12:47.355 INFO:teuthology.orchestra.run.smithi067.stdout:5 actionable tasks: 5 executed 2019-12-09T21:12:47.358 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:12:47.476 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.460+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.509 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.491+0000 7f1b734cfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:47.509 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.491+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.516 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.499+0000 7f1b734cfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:47.517 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.499+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.520 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.504+0000 7f1b734cfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:47.520 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.504+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.534 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.519+0000 7f1b734cfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:47.693 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.677+0000 7f1b734cfec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:12:47.693 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.677+0000 7f1b734cfec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:12:47.693 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.677+0000 7f1b734cfec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:12:47.694 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.677+0000 7f1b734cfec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:12:47.694 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.677+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.687+0000 7f1b734cfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:47.703 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.687+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.696+0000 7f1b734cfec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:12:47.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.696+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.719 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.703+0000 7f1b734cfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:47.719 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.703+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.711+0000 7f1b734cfec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:12:47.727 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.711+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.716+0000 7f1b734cfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:47.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.716+0000 7f1b734cfec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:12:47.732 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.716+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.720+0000 7f1b734cfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:47.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.720+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.731+0000 7f1b734cfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:47.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.731+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.742+0000 7f1b734cfec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:12:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.742+0000 7f1b734cfec0 20 zonegroup default 2019-12-09T21:12:47.758 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.742+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.747+0000 7f1b734cfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.747+0000 7f1b734cfec0 10 Cannot find current period zone using local zone 2019-12-09T21:12:47.763 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.747+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.753+0000 7f1b734cfec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:47.769 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.753+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.779 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.762+0000 7f1b734cfec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:47.779 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.762+0000 7f1b734cfec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:47.787 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.771+0000 7f1b734cfec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:12:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.771+0000 7f1b734cfec0 20 zone default found 2019-12-09T21:12:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.771+0000 7f1b734cfec0 20 started sync module instance, tier type = 2019-12-09T21:12:47.788 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:47.771+0000 7f1b734cfec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:12:48.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.098+0000 7f1b606f9700 0 -- 172.21.15.67:0/2631084883 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x5560e8074fa0 msgr2=0x5560e80773c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:12:48.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.167+0000 7f1b734cfec0 20 add_watcher() i=0 2019-12-09T21:12:48.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.167+0000 7f1b734cfec0 20 add_watcher() i=1 2019-12-09T21:12:48.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.167+0000 7f1b734cfec0 20 add_watcher() i=2 2019-12-09T21:12:48.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.167+0000 7f1b734cfec0 20 add_watcher() i=3 2019-12-09T21:12:48.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.167+0000 7f1b734cfec0 20 add_watcher() i=4 2019-12-09T21:12:48.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.167+0000 7f1b734cfec0 20 add_watcher() i=5 2019-12-09T21:12:48.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.167+0000 7f1b734cfec0 20 add_watcher() i=6 2019-12-09T21:12:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.205+0000 7f1b734cfec0 20 add_watcher() i=7 2019-12-09T21:12:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.205+0000 7f1b734cfec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:12:48.229 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.205+0000 7f1b30fe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:12:48.242 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:48.219+0000 7f1b60efa700 0 -- >> v1:172.21.15.81:6789/0 conn(0x5560e808e820 legacy=0x5560e808dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:12:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.708+0000 7f1aa2ffd700 20 reqs_thread_entry: start 2019-12-09T21:12:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.708+0000 7f1aa0ff9700 20 reqs_thread_entry: start 2019-12-09T21:12:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.708+0000 7f1b734cfec0 20 init_complete bucket index max shards: 0 2019-12-09T21:12:49.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.708+0000 7f1a96ffd700 20 reqs_thread_entry: start 2019-12-09T21:12:49.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.708+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:12:49.796 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.780+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:49.821 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.805+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:49.838 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.822+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:49.901 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.885+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:12:49.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.948+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:49.987 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.970+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:50.003 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:49.987+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.069 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.053+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:12:50.120 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.103+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.136 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.120+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:50.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.133+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.189 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.173+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:12:50.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.231+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.268 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.252+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:50.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.267+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.321 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.301+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:12:50.363 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.347+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.378 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.362+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:50.388 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.371+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.420 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.403+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:12:50.484 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.468+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.513 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.490+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.503+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.698+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:12:50.819 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.803+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.835 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.819+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:50.844 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.828+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.882 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.866+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:12:50.948 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.932+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:50.967 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.951+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:50.977 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:50.960+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.021 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.004+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:12:51.063 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.047+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.078 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.063+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:51.087 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.072+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.119 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.104+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:12:51.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.151+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.183 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.167+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:51.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.177+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.234 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.219+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:12:51.311 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.295+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.331 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.315+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:51.339 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.324+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.388 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.372+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:12:51.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.448+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.494 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.478+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:51.520 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.502+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.570 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.554+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:12:51.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.612+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.636+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:51.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.651+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.719 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.703+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:12:51.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.785+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.827 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.811+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:51.840 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.824+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.882 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.866+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:12:51.927 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.909+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.941 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.925+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:51.950 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.934+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:51.983 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:51.967+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:12:52.027 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.011+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.044 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.027+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:52.062 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.036+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.080 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.064+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:12:52.162 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.146+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.193 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.176+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:52.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.190+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.253 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.237+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:12:52.322 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.306+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.326+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:52.356 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.339+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.428 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.412+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:12:52.530 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.514+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.550 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.534+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:52.559 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.543+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.589+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:12:52.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.644+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.688 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.672+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:52.698 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.683+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.752 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.736+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:12:52.825 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.810+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.855 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.839+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:52.867 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.850+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.911 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.895+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:12:52.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.949+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:52.980 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.964+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:52.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:52.973+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.038+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:12:53.117 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.101+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.153 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.133+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:53.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.155+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.233 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.217+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:12:53.296 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.280+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.302+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:53.338 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.312+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.376 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.360+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:12:53.445 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.429+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.487 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.463+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:53.494 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.477+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.546 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.529+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:12:53.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.579+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.604+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:53.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.616+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.658+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:12:53.734 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.718+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.759 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.742+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:53.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.750+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.816 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.799+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:12:53.867 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.851+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.874+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:53.901 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.886+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:53.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.928+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:12:53.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:53.983+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:54.025 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.008+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:54.036 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.021+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:54.079 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.064+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:12:54.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.110+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:54.146 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.128+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:54.157 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.137+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:54.201 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.180+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:12:54.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.228+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:54.263 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.247+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:54.296 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.276+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:54.333 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.311+0000 7f1b734cfec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:12:54.364 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.348+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:54.382 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.364+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:54.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.372+0000 7f1b734cfec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:54.482 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.466+0000 7f1b734cfec0 20 remove_watcher() i=0 2019-12-09T21:12:54.482 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.466+0000 7f1b734cfec0 2 removed watcher, disabling cache 2019-12-09T21:12:54.528 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.512+0000 7f1b734cfec0 20 remove_watcher() i=1 2019-12-09T21:12:54.571 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.555+0000 7f1b734cfec0 20 remove_watcher() i=2 2019-12-09T21:12:54.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.611+0000 7f1b734cfec0 20 remove_watcher() i=3 2019-12-09T21:12:54.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.654+0000 7f1b734cfec0 20 remove_watcher() i=4 2019-12-09T21:12:54.709 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.694+0000 7f1b734cfec0 20 remove_watcher() i=5 2019-12-09T21:12:54.769 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.753+0000 7f1b734cfec0 20 remove_watcher() i=6 2019-12-09T21:12:54.813 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.797+0000 7f1b734cfec0 20 remove_watcher() i=7 2019-12-09T21:12:54.824 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:12:54.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.947+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:54.987 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.972+0000 7f1d9bf2cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:54.988 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.972+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:54.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.982+0000 7f1d9bf2cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:55.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.982+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.002 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.986+0000 7f1d9bf2cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:55.064 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.986+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.065 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:54.994+0000 7f1d9bf2cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:55.207 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.186+0000 7f1d9bf2cec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:12:55.207 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.186+0000 7f1d9bf2cec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:12:55.207 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.186+0000 7f1d9bf2cec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:12:55.207 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.186+0000 7f1d9bf2cec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:12:55.208 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.186+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.214 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.197+0000 7f1d9bf2cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:55.214 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.197+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.224 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.208+0000 7f1d9bf2cec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:12:55.224 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.208+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.232 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.216+0000 7f1d9bf2cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:55.233 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.216+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.242 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.226+0000 7f1d9bf2cec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:12:55.243 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.227+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.231+0000 7f1d9bf2cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:55.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.231+0000 7f1d9bf2cec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:12:55.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.231+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.252 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.236+0000 7f1d9bf2cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:55.252 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.236+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.244+0000 7f1d9bf2cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:55.260 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.244+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.253+0000 7f1d9bf2cec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:12:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.253+0000 7f1d9bf2cec0 20 zonegroup default 2019-12-09T21:12:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.253+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.258+0000 7f1d9bf2cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:55.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.258+0000 7f1d9bf2cec0 10 Cannot find current period zone using local zone 2019-12-09T21:12:55.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.258+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.280 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.264+0000 7f1d9bf2cec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:12:55.280 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.264+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.273+0000 7f1d9bf2cec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:12:55.289 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.273+0000 7f1d9bf2cec0 20 rados->read ofs=0 len=0 2019-12-09T21:12:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.283+0000 7f1d9bf2cec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:12:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.283+0000 7f1d9bf2cec0 20 zone default found 2019-12-09T21:12:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.283+0000 7f1d9bf2cec0 20 started sync module instance, tier type = 2019-12-09T21:12:55.299 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.283+0000 7f1d9bf2cec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:12:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.777+0000 7f1d9bf2cec0 20 add_watcher() i=0 2019-12-09T21:12:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.777+0000 7f1d9bf2cec0 20 add_watcher() i=1 2019-12-09T21:12:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.777+0000 7f1d9bf2cec0 20 add_watcher() i=2 2019-12-09T21:12:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.778+0000 7f1d9bf2cec0 20 add_watcher() i=3 2019-12-09T21:12:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.778+0000 7f1d9bf2cec0 20 add_watcher() i=4 2019-12-09T21:12:55.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.778+0000 7f1d9bf2cec0 20 add_watcher() i=5 2019-12-09T21:12:55.798 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.778+0000 7f1d9bf2cec0 20 add_watcher() i=6 2019-12-09T21:12:55.842 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.824+0000 7f1d9bf2cec0 20 add_watcher() i=7 2019-12-09T21:12:55.842 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.824+0000 7f1d9bf2cec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:12:55.842 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.824+0000 7f1d59fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:12:55.845 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.830+0000 7f1d89957700 0 -- 172.21.15.67:0/721568284 >> v2:172.21.15.67:3300/0 conn(0x564a643ef3f0 msgr2=0x564a64411500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T21:12:55.909 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:55.893+0000 7f1d89957700 0 -- 172.21.15.67:0/3019101513 >> [v2:172.21.15.67:6800/9311,v1:172.21.15.67:6801/9311] conn(0x7f1ccc0224c0 msgr2=0x7f1ccc024990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-09T21:12:57.359 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.342+0000 7f1ccbfff700 20 reqs_thread_entry: start 2019-12-09T21:12:57.359 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.342+0000 7f1cc9ffb700 20 reqs_thread_entry: start 2019-12-09T21:12:57.359 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.342+0000 7f1d9bf2cec0 20 init_complete bucket index max shards: 0 2019-12-09T21:12:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.343+0000 7f1cb7fff700 20 reqs_thread_entry: start 2019-12-09T21:12:57.360 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.343+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:12:57.422 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.405+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:57.444 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.427+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:57.461 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.436+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:57.492 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.474+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:12:57.558 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.534+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:57.573 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.557+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:57.582 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.565+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:57.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.608+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:12:57.686 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.669+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:57.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.689+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:57.716 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.700+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:57.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.743+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:12:57.824 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.807+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:57.848 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.831+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:57.861 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.844+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:57.949 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.894+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:12:57.957 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.938+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:57.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.953+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:57.979 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.962+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.013 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:57.996+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:12:58.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.065+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.111 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.094+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:58.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.110+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.185 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.163+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:12:58.263 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.245+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.298 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.281+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:58.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.292+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.327+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:12:58.385 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.368+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.403 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.386+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:58.412 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.395+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.442 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.425+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:12:58.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.486+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.531 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.514+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:58.542 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.525+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.588+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:12:58.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.655+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.683+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:58.718 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.701+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.742+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:12:58.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.822+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.867 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.850+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:58.880 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.864+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:58.941 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.916+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:12:59.019 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:58.991+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.044 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.021+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:59.058 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.039+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.111 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.094+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:12:59.185 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.163+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.188+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:59.217 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.200+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.244+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:12:59.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.302+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.328+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:59.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.338+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.564 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.535+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:12:59.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.612+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.647+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:59.682 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.663+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.732 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.713+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:12:59.794 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.777+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.811 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.794+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:59.820 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.803+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.868 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.851+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:12:59.947 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.930+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:12:59.974 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.957+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:12:59.992 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:12:59.975+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.211 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.193+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:13:00.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.245+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.296 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.263+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:00.296 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.274+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.323 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.305+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:13:00.400 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.383+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.430 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.413+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:00.446 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.429+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.488+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:13:00.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.550+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.590 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.573+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:00.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.587+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.634+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:13:00.710 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.693+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.724+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:00.752 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.735+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.773+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:13:00.851 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.830+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.881 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.848+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:00.882 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.858+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:00.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.897+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:13:00.985 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.966+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.988+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:01.025 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:00.999+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.080 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.063+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:13:01.140 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.123+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.160 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.143+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:01.167 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.150+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.214 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.196+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:13:01.266 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.249+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.293 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.275+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:01.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.296+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.361 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.345+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:13:01.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.388+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.427 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.411+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:01.442 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.425+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.470 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.454+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:13:01.529 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.511+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.547 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.530+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:01.557 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.540+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.591+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:13:01.691 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.674+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.713 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.696+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:01.753 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.731+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.787+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:13:01.864 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.847+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.884 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.864+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:01.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.873+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:01.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.922+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:13:02.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.979+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:02.012 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:01.994+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:02.020 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.003+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:02.064 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.047+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:13:02.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.096+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:02.148 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.122+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:02.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.131+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:02.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.177+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:13:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.234+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:02.279 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.262+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:02.306 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.277+0000 7f1d9bf2cec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:02.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.389+0000 7f1d9bf2cec0 20 remove_watcher() i=0 2019-12-09T21:13:02.407 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.389+0000 7f1d9bf2cec0 2 removed watcher, disabling cache 2019-12-09T21:13:02.452 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.435+0000 7f1d9bf2cec0 20 remove_watcher() i=1 2019-12-09T21:13:02.518 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.494+0000 7f1d9bf2cec0 20 remove_watcher() i=2 2019-12-09T21:13:02.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.577+0000 7f1d9bf2cec0 20 remove_watcher() i=3 2019-12-09T21:13:02.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.636+0000 7f1d9bf2cec0 20 remove_watcher() i=4 2019-12-09T21:13:02.696 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.679+0000 7f1d9bf2cec0 20 remove_watcher() i=5 2019-12-09T21:13:02.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.722+0000 7f1d9bf2cec0 20 remove_watcher() i=6 2019-12-09T21:13:02.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.755+0000 7f1d9bf2cec0 20 remove_watcher() i=7 2019-12-09T21:13:02.780 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:13:02.899 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.883+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.908+0000 7f72cd2d6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:02.925 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.908+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:02.933 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.916+0000 7f72cd2d6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:02.933 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.916+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.921+0000 7f72cd2d6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:02.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.921+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:02.945 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:02.929+0000 7f72cd2d6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:03.122 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.104+0000 7f72cd2d6ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:13:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.104+0000 7f72cd2d6ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:13:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.104+0000 7f72cd2d6ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:13:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.104+0000 7f72cd2d6ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:13:03.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.104+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.136 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.116+0000 7f72cd2d6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:03.137 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.116+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.144 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.128+0000 7f72cd2d6ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:13:03.145 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.128+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.138+0000 7f72cd2d6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.138+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.149+0000 7f72cd2d6ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:13:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.149+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.154+0000 7f72cd2d6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.154+0000 7f72cd2d6ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:13:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.154+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.159+0000 7f72cd2d6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.159+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.186 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.169+0000 7f72cd2d6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:03.186 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.169+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.177+0000 7f72cd2d6ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:13:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.177+0000 7f72cd2d6ec0 20 zonegroup default 2019-12-09T21:13:03.195 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.177+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.199 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.183+0000 7f72cd2d6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:03.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.183+0000 7f72cd2d6ec0 10 Cannot find current period zone using local zone 2019-12-09T21:13:03.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.183+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.205 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.188+0000 7f72cd2d6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:03.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.188+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.212 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.196+0000 7f72cd2d6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:03.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.196+0000 7f72cd2d6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.204+0000 7f72cd2d6ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:13:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.204+0000 7f72cd2d6ec0 20 zone default found 2019-12-09T21:13:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.204+0000 7f72cd2d6ec0 20 started sync module instance, tier type = 2019-12-09T21:13:03.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.204+0000 7f72cd2d6ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:13:03.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.619+0000 7f72cd2d6ec0 20 add_watcher() i=0 2019-12-09T21:13:03.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.619+0000 7f72cd2d6ec0 20 add_watcher() i=1 2019-12-09T21:13:03.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.619+0000 7f72cd2d6ec0 20 add_watcher() i=2 2019-12-09T21:13:03.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.619+0000 7f72cd2d6ec0 20 add_watcher() i=3 2019-12-09T21:13:03.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.619+0000 7f72cd2d6ec0 20 add_watcher() i=4 2019-12-09T21:13:03.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.619+0000 7f72cd2d6ec0 20 add_watcher() i=5 2019-12-09T21:13:03.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.619+0000 7f72cd2d6ec0 20 add_watcher() i=6 2019-12-09T21:13:03.686 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.668+0000 7f72cd2d6ec0 20 add_watcher() i=7 2019-12-09T21:13:03.686 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.668+0000 7f72cd2d6ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:13:03.686 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.668+0000 7f728afdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:13:03.688 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.671+0000 7f72bad01700 0 -- 172.21.15.67:0/1092724092 >> v1:172.21.15.81:6789/0 conn(0x561e91fb0320 legacy=0x561e91fd17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send injecting socket failure 2019-12-09T21:13:03.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:03.709+0000 7f72bad01700 0 -- 172.21.15.67:0/4195444110 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x561e91fcf320 msgr2=0x561e91fce6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:13:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.144+0000 7f71f97fa700 20 reqs_thread_entry: start 2019-12-09T21:13:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.144+0000 7f71f2ffd700 20 reqs_thread_entry: start 2019-12-09T21:13:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.144+0000 7f72cd2d6ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:13:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.145+0000 7f71f17fa700 20 reqs_thread_entry: start 2019-12-09T21:13:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.145+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:13:05.224 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.206+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.249 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.229+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:05.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.240+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.273+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:13:05.361 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.334+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.350+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.362+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.428 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.411+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:13:05.513 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.496+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.548 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.531+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:05.566 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.549+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.625+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:13:05.700 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.683+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.721 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.704+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:05.735 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.718+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.775 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.758+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:13:05.849 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.832+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.877 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.861+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:05.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.872+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:05.945 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.928+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:13:06.008 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:05.992+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.017+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:06.048 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.031+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.109 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.091+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:13:06.165 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.148+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.181 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.165+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:06.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.174+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.207+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:13:06.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.287+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.335 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.318+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:06.354 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.338+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.390+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:13:06.482 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.465+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.508 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.491+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:06.517 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.500+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.576 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.559+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:13:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.642+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.658+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.670+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.729 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.713+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:13:06.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.778+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.817 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.801+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:06.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.813+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.870 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.853+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:13:06.934 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.917+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:06.952 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.935+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:06.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.948+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.003 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:06.986+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:13:07.070 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.054+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.071+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:07.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.080+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.131 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.115+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:13:07.181 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.164+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.183+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.193+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.231+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:13:07.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.296+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.331 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.314+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:07.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.324+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.375+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:13:07.484 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.467+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.497+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:07.525 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.508+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.582 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.565+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:13:07.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.615+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.635+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:07.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.645+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.688+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:13:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.768+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.806+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.829+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.893 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.877+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:13:07.953 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.936+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:07.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.952+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:07.979 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:07.960+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.001+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:13:08.076 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.059+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.093 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.075+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:08.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.084+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.148 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.131+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:13:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.183+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.200+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.210+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.262 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.245+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:13:08.316 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.297+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.329 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.313+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:08.338 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.321+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.371 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.355+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:13:08.413 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.396+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.428 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.411+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:08.437 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.420+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.475 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.458+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:13:08.549 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.530+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.579 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.562+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:08.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.574+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.628+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:13:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.710+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.754 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.736+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:08.769 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.752+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.791+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:13:08.892 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.874+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.909 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.892+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:08.918 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.901+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:08.968 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:08.952+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:13:09.044 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.027+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.080+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:09.111 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.094+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.133+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:13:09.202 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.185+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.233 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.201+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:09.233 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.210+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.273 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.252+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:13:09.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.301+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.334 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.316+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:09.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.325+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.372+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:13:09.469 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.452+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.488 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.471+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:09.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.483+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.555 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.538+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:13:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.622+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.639+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.650+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.726+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:13:09.793 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.771+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.808 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.788+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:09.817 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.798+0000 7f72cd2d6ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:09.918 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.900+0000 7f72cd2d6ec0 20 remove_watcher() i=0 2019-12-09T21:13:09.918 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.900+0000 7f72cd2d6ec0 2 removed watcher, disabling cache 2019-12-09T21:13:09.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.950+0000 7f72cd2d6ec0 20 remove_watcher() i=1 2019-12-09T21:13:10.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:09.990+0000 7f72cd2d6ec0 20 remove_watcher() i=2 2019-12-09T21:13:10.084 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.067+0000 7f72cd2d6ec0 20 remove_watcher() i=3 2019-12-09T21:13:10.134 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.117+0000 7f72cd2d6ec0 20 remove_watcher() i=4 2019-12-09T21:13:10.185 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.168+0000 7f72cd2d6ec0 20 remove_watcher() i=5 2019-12-09T21:13:10.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.227+0000 7f72cd2d6ec0 20 remove_watcher() i=6 2019-12-09T21:13:10.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.307+0000 7f72cd2d6ec0 20 remove_watcher() i=7 2019-12-09T21:13:10.335 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:13:10.473 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.455+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.499 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.479+0000 7f4171905ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:10.500 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.479+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.489+0000 7f4171905ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:10.506 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.489+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.493+0000 7f4171905ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:10.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.493+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.521 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.504+0000 7f4171905ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.709+0000 7f4171905ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:13:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.709+0000 7f4171905ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:13:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.709+0000 7f4171905ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:13:10.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.709+0000 7f4171905ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:13:10.727 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.709+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.718+0000 7f4171905ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:10.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.718+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.743 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.726+0000 7f4171905ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:13:10.743 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.726+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.738+0000 7f4171905ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:10.755 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.738+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.762 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.745+0000 7f4171905ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:13:10.763 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.745+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.770 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.752+0000 7f4171905ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:10.770 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.752+0000 7f4171905ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:13:10.770 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.752+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.775 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.757+0000 7f4171905ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:10.775 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.757+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.765+0000 7f4171905ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:10.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.765+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.790 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.773+0000 7f4171905ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:13:10.790 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.773+0000 7f4171905ec0 20 zonegroup default 2019-12-09T21:13:10.790 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.773+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.777+0000 7f4171905ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:10.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.777+0000 7f4171905ec0 10 Cannot find current period zone using local zone 2019-12-09T21:13:10.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.777+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.800 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.782+0000 7f4171905ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:13:10.800 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.782+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.814 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.796+0000 7f4171905ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:13:10.815 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.796+0000 7f4171905ec0 20 rados->read ofs=0 len=0 2019-12-09T21:13:10.823 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.806+0000 7f4171905ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:13:10.823 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.806+0000 7f4171905ec0 20 zone default found 2019-12-09T21:13:10.823 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.806+0000 7f4171905ec0 20 started sync module instance, tier type = 2019-12-09T21:13:10.824 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:10.806+0000 7f4171905ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:13:11.250 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.233+0000 7f4171905ec0 20 add_watcher() i=0 2019-12-09T21:13:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.233+0000 7f4171905ec0 20 add_watcher() i=1 2019-12-09T21:13:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.233+0000 7f4171905ec0 20 add_watcher() i=2 2019-12-09T21:13:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.233+0000 7f4171905ec0 20 add_watcher() i=3 2019-12-09T21:13:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.233+0000 7f4171905ec0 20 add_watcher() i=4 2019-12-09T21:13:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.233+0000 7f4171905ec0 20 add_watcher() i=5 2019-12-09T21:13:11.251 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.233+0000 7f4171905ec0 20 add_watcher() i=6 2019-12-09T21:13:11.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.291+0000 7f4171905ec0 20 add_watcher() i=7 2019-12-09T21:13:11.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.291+0000 7f4171905ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:13:11.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.291+0000 7f412f7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:13:11.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.298+0000 7f415e32e700 0 -- 172.21.15.67:0/1498702027 >> v2:172.21.15.67:3300/0 conn(0x558b7ce48520 msgr2=0x558b7ce4a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T21:13:11.386 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:11.369+0000 7f415f330700 0 -- 172.21.15.67:0/3546793783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f40a0001b00 msgr2=0x7f40a00041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:13:13.039 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.021+0000 7f409dffb700 20 reqs_thread_entry: start 2019-12-09T21:13:13.039 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.022+0000 7f40977fe700 20 reqs_thread_entry: start 2019-12-09T21:13:13.039 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.022+0000 7f4171905ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:13:13.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.022+0000 7f4095ffb700 20 reqs_thread_entry: start 2019-12-09T21:13:13.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.022+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:13:13.104 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.087+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.131 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.114+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.127+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.177+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:13:13.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.224+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.257 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.240+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:13.267 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.249+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.302 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.285+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:13:13.346 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.329+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.361 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.344+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.353+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.388+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:13:13.463 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.446+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.483 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.467+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:13.493 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.476+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.532 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.514+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:13:13.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.608+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.646+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:13.680 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.662+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.728 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.711+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:13:13.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.779+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.812+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:13.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.822+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.861+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:13:13.946 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.929+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:13.979 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.953+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:13.980 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:13.964+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.029 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.010+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:13:14.093 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.076+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.119 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.102+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:14.132 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.114+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.170 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.152+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:13:14.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.199+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.233 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.215+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:14.241 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.225+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.260+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:13:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.324+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.339+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:14.363 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.346+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.409 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.393+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:13:14.459 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.442+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.474 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.457+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:14.486 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.469+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.531 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.514+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:13:14.582 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.565+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.583+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:14.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.591+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.642+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:13:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.690+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.719+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:14.752 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.735+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.770+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:13:14.833 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.817+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.835+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:14.862 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.843+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.901 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.883+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:13:14.953 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.936+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:14.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.953+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:14.978 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:14.961+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.007+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:13:15.106 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.089+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.142 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.125+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:15.161 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.143+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.209+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:13:15.286 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.269+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.286+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:15.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.296+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.347 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.330+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:13:15.462 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.445+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.508 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.489+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:15.523 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.507+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.576 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.559+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:13:15.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.604+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.623+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:15.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.632+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.670+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:13:15.734 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.717+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.753 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.734+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:15.764 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.747+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.796 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.779+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:13:15.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.871+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.921 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.904+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:15.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.921+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:15.997 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:15.980+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:13:16.061 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.041+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.084 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.068+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:16.098 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.082+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.146+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:13:16.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.222+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.266 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.249+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:16.276 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.259+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.333 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.314+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:13:16.420 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.403+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.446 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.429+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:16.459 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.442+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.504 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.487+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:13:16.575 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.558+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.587+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:16.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.601+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.647+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:13:16.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.708+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.746 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.729+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.738+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.811 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.794+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:13:16.867 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.851+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.891 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.874+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:16.902 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.885+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:16.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.919+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:13:16.988 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.971+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.003 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.987+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:17.014 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:16.997+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.054 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.037+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:13:17.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.078+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.118 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.101+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:17.128 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.111+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.167 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.151+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:13:17.210 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.194+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.242 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.225+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:17.253 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.237+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.286 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.269+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:13:17.348 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.331+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.381 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.364+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:17.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.375+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.433 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.416+0000 7f4171905ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:13:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.460+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.494 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.477+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:13:17.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.488+0000 7f4171905ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:13:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.562+0000 7f4171905ec0 20 remove_watcher() i=0 2019-12-09T21:13:17.579 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.562+0000 7f4171905ec0 2 removed watcher, disabling cache 2019-12-09T21:13:17.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.600+0000 7f4171905ec0 20 remove_watcher() i=1 2019-12-09T21:13:17.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.638+0000 7f4171905ec0 20 remove_watcher() i=2 2019-12-09T21:13:17.700 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.683+0000 7f4171905ec0 20 remove_watcher() i=3 2019-12-09T21:13:17.762 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.745+0000 7f4171905ec0 20 remove_watcher() i=4 2019-12-09T21:13:17.799 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.781+0000 7f4171905ec0 20 remove_watcher() i=5 2019-12-09T21:13:17.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.813+0000 7f4171905ec0 20 remove_watcher() i=6 2019-12-09T21:13:17.860 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:13:17.843+0000 7f4171905ec0 20 remove_watcher() i=7 2019-12-09T21:13:17.866 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle-4.7/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest 2019-12-09T21:13:18.492 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :clean 2019-12-09T21:13:18.989 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :compileJava 2019-12-09T21:13:18.989 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :processResources 2019-12-09T21:13:18.990 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :classes 2019-12-09T21:13:19.189 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:19.189 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :compileTestJava 2019-12-09T21:13:19.190 INFO:teuthology.orchestra.run.smithi067.stdout:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2019-12-09T21:13:19.190 INFO:teuthology.orchestra.run.smithi067.stdout:Note: Recompile with -Xlint:deprecation for details. 2019-12-09T21:13:19.190 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:19.190 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :processTestResources NO-SOURCE 2019-12-09T21:13:19.191 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :testClasses 2019-12-09T21:13:21.688 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:21.689 INFO:teuthology.orchestra.run.smithi067.stdout:> Task :test 2019-12-09T21:13:21.689 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:21.689 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.TestObjectCopyKeyNotFound STARTED 2019-12-09T21:13:25.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:25.889 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.TestObjectCopyKeyNotFound PASSED 2019-12-09T21:13:35.489 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:35.489 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadExpectNone STARTED 2019-12-09T21:13:38.788 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:38.788 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadExpectNone PASSED 2019-12-09T21:13:45.487 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:45.487 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadMd5Empty STARTED 2019-12-09T21:13:47.887 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:47.887 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.TestObjectCreateBadMd5Empty PASSED 2019-12-09T21:13:51.986 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:13:51.987 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadLLAPI STARTED 2019-12-09T21:14:17.185 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:17.185 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadLLAPI PASSED 2019-12-09T21:14:22.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:22.085 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadNotFoundLLAPI STARTED 2019-12-09T21:14:24.285 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:24.288 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testAbortMultipartUploadNotFoundLLAPI PASSED 2019-12-09T21:14:29.285 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:29.285 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testDownloadHLAPI STARTED 2019-12-09T21:14:32.784 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:32.784 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testDownloadHLAPI PASSED 2019-12-09T21:14:40.785 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:40.785 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testDownloadNoBucketHLAPI STARTED 2019-12-09T21:14:41.084 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:41.084 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testDownloadNoBucketHLAPI PASSED 2019-12-09T21:14:42.084 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:42.084 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testDownloadNoKeyHLAPI STARTED 2019-12-09T21:14:45.785 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:45.785 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testDownloadNoKeyHLAPI PASSED 2019-12-09T21:14:51.983 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:51.983 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1MB STARTED 2019-12-09T21:14:56.784 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:14:56.784 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1MB PASSED 2019-12-09T21:15:03.383 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:03.383 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1b STARTED 2019-12-09T21:15:06.182 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:06.182 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1b PASSED 2019-12-09T21:15:13.285 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:13.285 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1kb STARTED 2019-12-09T21:15:18.482 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:18.482 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptedTransfer1kb PASSED 2019-12-09T21:15:25.881 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:25.881 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptionKeyNoSSEC STARTED 2019-12-09T21:15:28.181 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:28.181 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptionKeyNoSSEC PASSED 2019-12-09T21:15:35.881 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:35.881 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECInvalidMd5 STARTED 2019-12-09T21:15:37.781 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:37.781 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECInvalidMd5 PASSED 2019-12-09T21:15:42.380 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:42.380 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoKey STARTED 2019-12-09T21:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:43.683 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoKey PASSED 2019-12-09T21:15:47.280 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:47.281 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoMd5 STARTED 2019-12-09T21:15:48.580 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:48.580 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testEncryptionKeySSECNoMd5 PASSED 2019-12-09T21:15:53.080 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:53.080 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultiObjectDelete STARTED 2019-12-09T21:15:56.380 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:56.380 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultiObjectDelete PASSED 2019-12-09T21:15:58.402 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:42:19.458 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:15:58.409 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:42:19.465 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:15:58.419 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:42:19.475 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:15:58.424 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:42:19.480 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:15:58.425 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:42:19.585 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:15:58.430 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:42:19.590 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:15:58.612 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:42:19.668 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:15:58.648 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:42:19.704 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:15:58.656 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:42:19.817 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:15:59.580 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:15:59.580 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartCopyHLAPIA STARTED 2019-12-09T21:16:12.354 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:42:33.411 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:16:12.377 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:42:33.435 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:16:12.395 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:42:33.451 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:16:12.397 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:42:33.455 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:16:12.398 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:42:33.455 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:16:12.411 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:42:33.571 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:16:12.416 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:42:33.576 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:16:12.428 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:42:33.483 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:16:12.441 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:42:33.601 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:16:14.381 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:16:14.381 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartCopyHLAPIA PASSED 2019-12-09T21:16:25.078 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:16:25.079 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartCopyMultipleSizesLLAPI STARTED 2019-12-09T21:19:54.472 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:19:54.473 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartCopyMultipleSizesLLAPI PASSED 2019-12-09T21:20:04.971 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:20:04.972 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCBucketHLAPI STARTED 2019-12-09T21:20:07.372 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:20:07.372 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCBucketHLAPI PASSED 2019-12-09T21:20:14.573 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:20:14.573 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCKeyHLAPI STARTED 2019-12-09T21:20:17.977 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:20:17.977 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartCopyNoSRCKeyHLAPI PASSED 2019-12-09T21:20:28.870 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:20:28.870 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartDownloadHLAPI STARTED 2019-12-09T21:20:41.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:20:41.674 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartDownloadHLAPI PASSED 2019-12-09T21:20:47.470 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:20:47.470 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartDownloadNoBucketHLAPI STARTED 2019-12-09T21:20:47.770 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:20:47.770 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartDownloadNoBucketHLAPI PASSED 2019-12-09T21:20:48.370 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:20:48.370 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartDownloadWithPauseHLAPI STARTED 2019-12-09T21:21:02.665 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T21:21:02.646+0000 41264700 -1 handle_data failed with exit code -5 2019-12-09T21:21:02.870 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:21:02.870 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartDownloadWithPauseHLAPI PASSED 2019-12-09T21:21:10.017 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:47:31.083 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:21:10.025 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:47:31.091 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:21:10.035 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:47:31.100 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:21:10.041 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:47:31.108 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:21:10.047 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:47:31.205 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:21:10.058 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:47:31.217 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:21:10.229 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:47:31.296 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:21:10.236 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:47:31.396 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:21:10.255 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:47:31.322 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:21:10.270 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:21:10.270 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadHLAPIAWS4 STARTED 2019-12-09T21:21:21.373 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:21:21.373 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadHLAPIAWS4 PASSED 2019-12-09T21:21:33.073 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:21:33.073 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadIncorrectMissingPartLLAPI STARTED 2019-12-09T21:21:55.369 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:21:55.370 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadIncorrectMissingPartLLAPI PASSED 2019-12-09T21:22:02.870 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:22:02.870 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadMultipleSizesLLAPI STARTED 2019-12-09T21:23:57.469 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:23:57.469 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadMultipleSizesLLAPI PASSED 2019-12-09T21:24:03.569 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:24:03.569 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadOverwriteExistingObjectLLAPI STARTED 2019-12-09T21:24:28.469 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:24:28.469 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadOverwriteExistingObjectLLAPI PASSED 2019-12-09T21:24:33.838 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:50:54.904 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:24:33.843 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:50:54.910 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:24:33.890 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:50:54.957 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:24:33.903 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:50:54.970 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:24:33.909 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:50:55.069 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:24:33.925 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:50:55.084 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:24:33.988 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:50:55.055 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:24:34.034 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:50:55.101 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:24:34.093 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:50:55.250 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:24:34.169 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:24:34.169 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadSmallLLAPI STARTED 2019-12-09T21:24:58.969 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:24:58.969 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadSmallLLAPI PASSED 2019-12-09T21:25:05.169 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:05.169 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadWithPause STARTED 2019-12-09T21:25:13.469 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:13.469 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testMultipartUploadWithPause PASSED 2019-12-09T21:25:26.269 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:26.269 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCopyBucketNotFound STARTED 2019-12-09T21:25:26.669 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:26.669 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCopyBucketNotFound PASSED 2019-12-09T21:25:27.569 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:27.569 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadContentlengthNegative STARTED 2019-12-09T21:25:29.269 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:29.269 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadContentlengthNegative PASSED 2019-12-09T21:25:32.969 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:32.969 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadExpectMismatch STARTED 2019-12-09T21:25:35.269 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:35.269 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadExpectMismatch PASSED 2019-12-09T21:25:39.369 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:39.369 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5InvalidShort STARTED 2019-12-09T21:25:40.569 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:40.569 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5InvalidShort PASSED 2019-12-09T21:25:43.569 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:43.569 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Ivalid STARTED 2019-12-09T21:25:45.669 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:45.669 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Ivalid PASSED 2019-12-09T21:25:48.669 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:48.669 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5None STARTED 2019-12-09T21:25:49.371 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:49.371 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5None PASSED 2019-12-09T21:25:52.069 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:52.069 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Unreadable STARTED 2019-12-09T21:25:53.769 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:53.769 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectCreateBadMd5Unreadable PASSED 2019-12-09T21:25:57.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:57.469 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectHeadZeroBytes STARTED 2019-12-09T21:25:59.769 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:25:59.769 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectHeadZeroBytes PASSED 2019-12-09T21:26:03.869 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:03.869 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterDot STARTED 2019-12-09T21:26:06.668 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:06.669 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterDot PASSED 2019-12-09T21:26:11.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:11.369 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterNotExist STARTED 2019-12-09T21:26:14.569 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:14.569 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterNotExist PASSED 2019-12-09T21:26:18.875 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:18.875 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterPercentage STARTED 2019-12-09T21:26:21.671 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:21.671 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListDelimiterPercentage PASSED 2019-12-09T21:26:22.136 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:52:43.203 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:26:22.140 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:52:43.208 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:26:22.150 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:52:43.218 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:26:22.152 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:52:43.311 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:26:22.154 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:52:43.222 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:26:22.156 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:52:43.316 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:26:22.228 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:52:43.295 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:26:22.243 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:52:43.402 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:26:22.253 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:52:43.321 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:26:26.268 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:26.268 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMarkerAfterList STARTED 2019-12-09T21:26:29.868 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:29.868 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMarkerAfterList PASSED 2019-12-09T21:26:36.268 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:36.268 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMarkerBeforeList STARTED 2019-12-09T21:26:43.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:43.069 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMarkerBeforeList PASSED 2019-12-09T21:26:54.968 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:26:54.968 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMarkerNotInList STARTED 2019-12-09T21:27:01.868 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:27:01.868 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMarkerNotInList PASSED 2019-12-09T21:27:12.668 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:27:12.668 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMarkerUnreadable STARTED 2019-12-09T21:27:20.668 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:27:20.668 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMarkerUnreadable PASSED 2019-12-09T21:27:29.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:27:29.068 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNegative STARTED 2019-12-09T21:27:35.268 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:27:35.268 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNegative PASSED 2019-12-09T21:27:47.167 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:27:47.168 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNone STARTED 2019-12-09T21:27:54.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:27:54.568 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysNone PASSED 2019-12-09T21:28:05.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:28:05.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysOne STARTED 2019-12-09T21:28:13.267 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:28:13.268 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysOne PASSED 2019-12-09T21:28:22.867 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:28:22.868 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysZero STARTED 2019-12-09T21:28:29.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:28:29.568 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListMaxkeysZero PASSED 2019-12-09T21:28:40.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:28:40.567 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixAlt STARTED 2019-12-09T21:28:46.167 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:28:46.167 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixAlt PASSED 2019-12-09T21:28:53.967 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:28:53.967 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixBasic STARTED 2019-12-09T21:28:59.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:28:59.469 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixBasic PASSED 2019-12-09T21:29:08.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:29:08.367 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterAlt STARTED 2019-12-09T21:29:17.167 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:29:17.167 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterAlt PASSED 2019-12-09T21:29:27.667 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:29:27.667 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterBasic STARTED 2019-12-09T21:29:34.267 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:29:34.267 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterBasic PASSED 2019-12-09T21:29:44.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:29:44.467 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterDelimiterNotExist STARTED 2019-12-09T21:29:52.667 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:29:52.667 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterDelimiterNotExist PASSED 2019-12-09T21:30:04.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:30:04.567 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixDelimiterNotExist STARTED 2019-12-09T21:30:09.366 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:30:09.367 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixDelimiterNotExist PASSED 2019-12-09T21:30:18.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:30:18.467 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixNotExist STARTED 2019-12-09T21:30:25.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:30:25.367 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixDelimiterPrefixNotExist PASSED 2019-12-09T21:30:37.867 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:30:37.867 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixEmpty STARTED 2019-12-09T21:30:42.277 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:30:42.277 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixEmpty PASSED 2019-12-09T21:30:54.267 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:30:54.267 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNone STARTED 2019-12-09T21:31:00.867 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:00.867 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNone PASSED 2019-12-09T21:31:10.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:10.368 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNotExist STARTED 2019-12-09T21:31:16.467 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:16.467 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixNotExist PASSED 2019-12-09T21:31:26.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:26.367 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixUnreadable STARTED 2019-12-09T21:31:31.867 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectListPrefixUnreadable PASSED 2019-12-09T21:31:34.478 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:57:55.546 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:31:34.483 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:57:55.552 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:31:34.491 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:57:55.650 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:31:34.493 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:57:55.562 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:31:34.497 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:57:55.655 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:31:34.500 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:57:55.569 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:31:34.668 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:57:55.737 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:31:34.687 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:57:55.845 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:31:34.759 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:57:55.827 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:31:41.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:41.367 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectReadFromNonExistantBucket STARTED 2019-12-09T21:31:42.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:42.673 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectReadFromNonExistantBucket PASSED 2019-12-09T21:31:49.167 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:49.168 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectReadNotExist STARTED 2019-12-09T21:31:52.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:52.368 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectReadNotExist PASSED 2019-12-09T21:31:58.367 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:31:58.368 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectWriteCacheControl STARTED 2019-12-09T21:32:02.567 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:02.568 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectWriteCacheControl PASSED 2019-12-09T21:32:11.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:11.168 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectWriteCheckEtag STARTED 2019-12-09T21:32:14.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:14.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectWriteCheckEtag PASSED 2019-12-09T21:32:20.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:20.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectWriteReadUpdateReadDelete STARTED 2019-12-09T21:32:27.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:27.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectWriteReadUpdateReadDelete PASSED 2019-12-09T21:32:31.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:31.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectWriteToNonExistantBucket STARTED 2019-12-09T21:32:31.868 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:31.868 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testObjectWriteToNonExistantBucket PASSED 2019-12-09T21:32:32.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:32.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testRangedSkipLeadingBytesResponseCode STARTED 2019-12-09T21:32:36.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:36.168 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testRangedSkipLeadingBytesResponseCode PASSED 2019-12-09T21:32:44.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:44.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testRangedrequestResponseCode STARTED 2019-12-09T21:32:47.568 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:47.569 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testRangedrequestResponseCode PASSED 2019-12-09T21:32:55.383 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:59:16.451 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:32:55.422 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:59:16.490 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:32:55.429 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:59:16.498 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:32:55.433 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:59:16.500 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:32:55.460 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:59:16.618 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:32:55.466 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:59:16.623 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:32:55.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:55.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testSSEKMSNoKey STARTED 2019-12-09T21:32:55.480 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:59:16.548 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:32:55.490 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:59:16.558 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:32:55.548 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:00:59:16.705 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:32:57.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:32:57.168 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testSSEKMSNoKey PASSED 2019-12-09T21:33:02.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:02.768 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testSSEKMSNotDeclared STARTED 2019-12-09T21:33:04.568 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:04.568 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testSSEKMSNotDeclared PASSED 2019-12-09T21:33:10.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:10.768 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testSSEKMSPresent STARTED 2019-12-09T21:33:14.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:14.368 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testSSEKMSPresent PASSED 2019-12-09T21:33:20.368 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:20.368 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testUploadFileHLAPIBigFile STARTED 2019-12-09T21:33:30.468 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:30.468 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testUploadFileHLAPIBigFile PASSED 2019-12-09T21:33:36.668 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:36.668 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testUploadFileListHLAPI STARTED 2019-12-09T21:33:48.068 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:48.068 INFO:teuthology.orchestra.run.smithi067.stdout:suite > Object tests > ObjectTest.testUploadFileListHLAPI PASSED 2019-12-09T21:33:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:54.906 INFO:teuthology.orchestra.run.smithi067.stdout:---------------------------------------------------------------------- 2019-12-09T21:33:54.906 INFO:teuthology.orchestra.run.smithi067.stdout:| Results: SUCCESS (72 tests, 72 successes, 0 failures, 0 skipped) | 2019-12-09T21:33:54.906 INFO:teuthology.orchestra.run.smithi067.stdout:---------------------------------------------------------------------- 2019-12-09T21:33:54.906 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:33:54.907 INFO:teuthology.orchestra.run.smithi067.stdout:BUILD SUCCESSFUL in 20m 36s 2019-12-09T21:33:54.907 INFO:teuthology.orchestra.run.smithi067.stdout:5 actionable tasks: 5 executed 2019-12-09T21:33:54.920 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:33:55.077 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.082+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.106+0000 7f059a35aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:33:55.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.106+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.110 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.114+0000 7f059a35aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:33:55.110 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.114+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.119+0000 7f059a35aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:33:55.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.119+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.122 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.127+0000 7f059a35aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:33:55.303 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.307+0000 7f059a35aec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:33:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.307+0000 7f059a35aec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:33:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.307+0000 7f059a35aec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:33:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.307+0000 7f059a35aec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:33:55.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.307+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.316+0000 7f059a35aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:33:55.312 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.316+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.323+0000 7f059a35aec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:33:55.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.323+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.328 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.332+0000 7f059a35aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:33:55.328 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.332+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.339+0000 7f059a35aec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:33:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.339+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.345+0000 7f059a35aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:33:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.345+0000 7f059a35aec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:33:55.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.345+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.354+0000 7f059a35aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:33:55.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.354+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.361+0000 7f059a35aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:33:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.361+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.369+0000 7f059a35aec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:33:55.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.369+0000 7f059a35aec0 20 zonegroup default 2019-12-09T21:33:55.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.369+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.370 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.374+0000 7f059a35aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:33:55.370 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.374+0000 7f059a35aec0 10 Cannot find current period zone using local zone 2019-12-09T21:33:55.370 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.374+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.375 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.380+0000 7f059a35aec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:33:55.376 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.380+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.389+0000 7f059a35aec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:33:55.385 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.389+0000 7f059a35aec0 20 rados->read ofs=0 len=0 2019-12-09T21:33:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.397+0000 7f059a35aec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:33:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.397+0000 7f059a35aec0 20 zone default found 2019-12-09T21:33:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.397+0000 7f059a35aec0 20 started sync module instance, tier type = 2019-12-09T21:33:55.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.397+0000 7f059a35aec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:33:55.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.783+0000 7f059a35aec0 20 add_watcher() i=0 2019-12-09T21:33:55.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.783+0000 7f059a35aec0 20 add_watcher() i=1 2019-12-09T21:33:55.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.783+0000 7f059a35aec0 20 add_watcher() i=2 2019-12-09T21:33:55.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.783+0000 7f059a35aec0 20 add_watcher() i=3 2019-12-09T21:33:55.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.783+0000 7f059a35aec0 20 add_watcher() i=4 2019-12-09T21:33:55.787 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.783+0000 7f059a35aec0 20 add_watcher() i=5 2019-12-09T21:33:55.787 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.787+0000 7f059a35aec0 20 add_watcher() i=6 2019-12-09T21:33:55.828 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.832+0000 7f059a35aec0 20 add_watcher() i=7 2019-12-09T21:33:55.829 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.832+0000 7f059a35aec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:33:55.829 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.832+0000 7f0557fdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:33:55.834 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.838+0000 7f0586d83700 0 -- 172.21.15.67:0/335503153 >> v2:172.21.15.67:3300/0 conn(0x55774c596b70 msgr2=0x55774c598f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T21:33:55.897 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:55.899+0000 7f0586d83700 0 -- 172.21.15.67:0/1291043254 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x55774c5a9be0 msgr2=0x55774c5ac000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:33:57.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.708+0000 7f04c67fc700 20 reqs_thread_entry: start 2019-12-09T21:33:57.705 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.709+0000 7f04bffff700 20 reqs_thread_entry: start 2019-12-09T21:33:57.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.709+0000 7f059a35aec0 20 init_complete bucket index max shards: 0 2019-12-09T21:33:57.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.709+0000 7f04be7fc700 20 reqs_thread_entry: start 2019-12-09T21:33:57.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.709+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:33:57.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.789+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:57.821 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.824+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:33:57.832 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.836+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:57.875 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.878+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:33:57.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.930+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:57.947 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.950+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:33:57.962 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:33:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1220', time=2019-12-09T23:23:10.888477+0000, chain.objs.size()=9 2019-12-09T21:33:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~XGXXIYXVJCnG-fylwX2q7sSRgXG_Uts.1 2019-12-09T21:33:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~XGXXIYXVJCnG-fylwX2q7sSRgXG_Uts.1_1 2019-12-09T21:33:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~XGXXIYXVJCnG-fylwX2q7sSRgXG_Uts.2 2019-12-09T21:33:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~XGXXIYXVJCnG-fylwX2q7sSRgXG_Uts.2_1 2019-12-09T21:33:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~XGXXIYXVJCnG-fylwX2q7sSRgXG_Uts.3 2019-12-09T21:33:57.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~XGXXIYXVJCnG-fylwX2q7sSRgXG_Uts.3_1 2019-12-09T21:33:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~XGXXIYXVJCnG-fylwX2q7sSRgXG_Uts.4 2019-12-09T21:33:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~XGXXIYXVJCnG-fylwX2q7sSRgXG_Uts.4_1 2019-12-09T21:33:57.964 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:57.965+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~XGXXIYXVJCnG-fylwX2q7sSRgXG_Uts.5 2019-12-09T21:33:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.228+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:33:58.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.348+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:33:58.388 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.392+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:58.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.407+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:33:58.416 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.419+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:33:58.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.419+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1232', time=2019-12-09T23:23:41.601640+0000, chain.objs.size()=7 2019-12-09T21:33:58.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.419+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~lyv_OgDPpdDJUkw6rEztqF0ijma9laf.1 2019-12-09T21:33:58.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.419+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~lyv_OgDPpdDJUkw6rEztqF0ijma9laf.1_1 2019-12-09T21:33:58.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.419+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~lyv_OgDPpdDJUkw6rEztqF0ijma9laf.1_2 2019-12-09T21:33:58.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.419+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~lyv_OgDPpdDJUkw6rEztqF0ijma9laf.2 2019-12-09T21:33:58.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.419+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~lyv_OgDPpdDJUkw6rEztqF0ijma9laf.2_1 2019-12-09T21:33:58.418 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.419+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~lyv_OgDPpdDJUkw6rEztqF0ijma9laf.2_2 2019-12-09T21:33:58.418 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.419+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~lyv_OgDPpdDJUkw6rEztqF0ijma9laf.3 2019-12-09T21:33:58.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.656+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:33:58.844 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.848+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:33:58.904 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.907+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:58.922 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.925+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:33:58.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.939+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:33:58.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.939+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1242', time=2019-12-09T23:23:58.803375+0000, chain.objs.size()=7 2019-12-09T21:33:58.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.939+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~S4RTiBDyf0J-eS8v75rXadxT3veBzIU.1 2019-12-09T21:33:58.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.939+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~S4RTiBDyf0J-eS8v75rXadxT3veBzIU.1_1 2019-12-09T21:33:58.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.939+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~S4RTiBDyf0J-eS8v75rXadxT3veBzIU.1_2 2019-12-09T21:33:58.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.939+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~S4RTiBDyf0J-eS8v75rXadxT3veBzIU.2 2019-12-09T21:33:58.937 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.939+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~S4RTiBDyf0J-eS8v75rXadxT3veBzIU.2_1 2019-12-09T21:33:58.937 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.939+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~S4RTiBDyf0J-eS8v75rXadxT3veBzIU.2_2 2019-12-09T21:33:58.937 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:58.939+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~S4RTiBDyf0J-eS8v75rXadxT3veBzIU.3 2019-12-09T21:33:59.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.104+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:33:59.245 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.248+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:33:59.307 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.311+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:59.333 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.336+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:33:59.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.354+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.411+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:33:59.513 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.517+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:59.555 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.558+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:33:59.565 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.569+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:59.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.618+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:33:59.695 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.699+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:59.724 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.727+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:33:59.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.742+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:59.796 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.800+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:33:59.853 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.856+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:33:59.873 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.877+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:33:59.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:33:59.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1181', time=2019-12-09T23:21:25.680937+0000, chain.objs.size()=9 2019-12-09T21:33:59.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file2.mpg.2~CeC1aD3OEOeHCTp8TcsGOqFklSchNrV.1 2019-12-09T21:33:59.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__shadow_file2.mpg.2~CeC1aD3OEOeHCTp8TcsGOqFklSchNrV.1_1 2019-12-09T21:33:59.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file2.mpg.2~CeC1aD3OEOeHCTp8TcsGOqFklSchNrV.2 2019-12-09T21:33:59.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__shadow_file2.mpg.2~CeC1aD3OEOeHCTp8TcsGOqFklSchNrV.2_1 2019-12-09T21:33:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file2.mpg.2~CeC1aD3OEOeHCTp8TcsGOqFklSchNrV.3 2019-12-09T21:33:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__shadow_file2.mpg.2~CeC1aD3OEOeHCTp8TcsGOqFklSchNrV.3_1 2019-12-09T21:33:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file2.mpg.2~CeC1aD3OEOeHCTp8TcsGOqFklSchNrV.4 2019-12-09T21:33:59.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__shadow_file2.mpg.2~CeC1aD3OEOeHCTp8TcsGOqFklSchNrV.4_1 2019-12-09T21:33:59.891 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:33:59.891+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file2.mpg.2~CeC1aD3OEOeHCTp8TcsGOqFklSchNrV.5 2019-12-09T21:34:00.220 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.223+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:00.331 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.334+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:34:00.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.377+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:00.390 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.394+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:00.401 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.405+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:00.401 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.405+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.968', time=2019-12-09T23:14:58.545516+0000, chain.objs.size()=2 2019-12-09T21:34:00.402 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.405+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.56__shadow_.JD9mPezD-D4wdI3pgkbdr6UG0SBBAiG_1 2019-12-09T21:34:00.402 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.405+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.56__shadow_.JD9mPezD-D4wdI3pgkbdr6UG0SBBAiG_2 2019-12-09T21:34:00.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:00.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.672+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:34:00.708 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.711+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:00.727 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.731+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:00.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.739+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:00.768 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.772+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:34:00.842 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.845+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:00.874 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.877+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:00.887 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.891+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:00.941 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:00.945+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:34:01.004 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.003+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:01.027 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.031+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1213', time=2019-12-09T23:22:50.005557+0000, chain.objs.size()=9 2019-12-09T21:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~Xv89W5RssXi6PJeuKzEeljqDcA2uhcy.1 2019-12-09T21:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~Xv89W5RssXi6PJeuKzEeljqDcA2uhcy.1_1 2019-12-09T21:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~Xv89W5RssXi6PJeuKzEeljqDcA2uhcy.2 2019-12-09T21:34:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~Xv89W5RssXi6PJeuKzEeljqDcA2uhcy.2_1 2019-12-09T21:34:01.054 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~Xv89W5RssXi6PJeuKzEeljqDcA2uhcy.3 2019-12-09T21:34:01.054 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~Xv89W5RssXi6PJeuKzEeljqDcA2uhcy.3_1 2019-12-09T21:34:01.054 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~Xv89W5RssXi6PJeuKzEeljqDcA2uhcy.4 2019-12-09T21:34:01.054 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~Xv89W5RssXi6PJeuKzEeljqDcA2uhcy.4_1 2019-12-09T21:34:01.054 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~Xv89W5RssXi6PJeuKzEeljqDcA2uhcy.5 2019-12-09T21:34:01.054 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1272', time=2019-12-09T23:25:01.138878+0000, chain.objs.size()=9 2019-12-09T21:34:01.054 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.77__multipart_key1.2~w0kazGQjxMBijP5Jl3BL8f7-O3Fvyr3.1 2019-12-09T21:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.77__shadow_key1.2~w0kazGQjxMBijP5Jl3BL8f7-O3Fvyr3.1_1 2019-12-09T21:34:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.056+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.77__multipart_key1.2~w0kazGQjxMBijP5Jl3BL8f7-O3Fvyr3.2 2019-12-09T21:34:01.268 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.271+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.77__shadow_key1.2~w0kazGQjxMBijP5Jl3BL8f7-O3Fvyr3.2_1 2019-12-09T21:34:01.268 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.271+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.77__multipart_key1.2~w0kazGQjxMBijP5Jl3BL8f7-O3Fvyr3.3 2019-12-09T21:34:01.268 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.271+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.77__shadow_key1.2~w0kazGQjxMBijP5Jl3BL8f7-O3Fvyr3.3_1 2019-12-09T21:34:01.268 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.271+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.77__multipart_key1.2~w0kazGQjxMBijP5Jl3BL8f7-O3Fvyr3.4 2019-12-09T21:34:01.268 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.271+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.77__shadow_key1.2~w0kazGQjxMBijP5Jl3BL8f7-O3Fvyr3.4_1 2019-12-09T21:34:01.302 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.306+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.77__multipart_key1.2~w0kazGQjxMBijP5Jl3BL8f7-O3Fvyr3.5 2019-12-09T21:34:01.555 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.559+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:01.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.671+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:34:01.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.716+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:01.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.742+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:34:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1068', time=2019-12-09T23:18:18.325018+0000, chain.objs.size()=9 2019-12-09T21:34:01.770 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VFb6TUSxDOdX4X0neonJ4Gh9C3rPDr4.1 2019-12-09T21:34:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~VFb6TUSxDOdX4X0neonJ4Gh9C3rPDr4.1_1 2019-12-09T21:34:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VFb6TUSxDOdX4X0neonJ4Gh9C3rPDr4.2 2019-12-09T21:34:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~VFb6TUSxDOdX4X0neonJ4Gh9C3rPDr4.2_1 2019-12-09T21:34:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VFb6TUSxDOdX4X0neonJ4Gh9C3rPDr4.3 2019-12-09T21:34:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~VFb6TUSxDOdX4X0neonJ4Gh9C3rPDr4.3_1 2019-12-09T21:34:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VFb6TUSxDOdX4X0neonJ4Gh9C3rPDr4.4 2019-12-09T21:34:01.771 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~VFb6TUSxDOdX4X0neonJ4Gh9C3rPDr4.4_1 2019-12-09T21:34:01.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VFb6TUSxDOdX4X0neonJ4Gh9C3rPDr4.5 2019-12-09T21:34:01.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1227', time=2019-12-09T23:23:26.059603+0000, chain.objs.size()=9 2019-12-09T21:34:01.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~QW1ZqiVbSPQ0x6BblVk2GNQcW9Y_l_7.1 2019-12-09T21:34:01.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~QW1ZqiVbSPQ0x6BblVk2GNQcW9Y_l_7.1_1 2019-12-09T21:34:01.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:01.773+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~QW1ZqiVbSPQ0x6BblVk2GNQcW9Y_l_7.2 2019-12-09T21:34:02.079 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.082+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~QW1ZqiVbSPQ0x6BblVk2GNQcW9Y_l_7.2_1 2019-12-09T21:34:02.106 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.110+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~QW1ZqiVbSPQ0x6BblVk2GNQcW9Y_l_7.3 2019-12-09T21:34:02.106 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.110+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~QW1ZqiVbSPQ0x6BblVk2GNQcW9Y_l_7.3_1 2019-12-09T21:34:02.168 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.172+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~QW1ZqiVbSPQ0x6BblVk2GNQcW9Y_l_7.4 2019-12-09T21:34:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.172+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~QW1ZqiVbSPQ0x6BblVk2GNQcW9Y_l_7.4_1 2019-12-09T21:34:02.169 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.172+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~QW1ZqiVbSPQ0x6BblVk2GNQcW9Y_l_7.5 2019-12-09T21:34:02.512 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.516+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:02.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.645+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:34:02.701 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.705+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:02.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.725+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:02.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.750+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:02.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.750+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1237', time=2019-12-09T23:23:56.453275+0000, chain.objs.size()=7 2019-12-09T21:34:02.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.750+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~25qa2wLbzVCJ2aKSxASAaqili15HvLy.1 2019-12-09T21:34:02.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.750+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~25qa2wLbzVCJ2aKSxASAaqili15HvLy.1_1 2019-12-09T21:34:02.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.750+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~25qa2wLbzVCJ2aKSxASAaqili15HvLy.1_2 2019-12-09T21:34:02.747 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.750+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~25qa2wLbzVCJ2aKSxASAaqili15HvLy.2 2019-12-09T21:34:02.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.750+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~25qa2wLbzVCJ2aKSxASAaqili15HvLy.2_1 2019-12-09T21:34:02.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.750+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__shadow_key1.2~25qa2wLbzVCJ2aKSxASAaqili15HvLy.2_2 2019-12-09T21:34:02.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:02.750+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.75__multipart_key1.2~25qa2wLbzVCJ2aKSxASAaqili15HvLy.3 2019-12-09T21:34:03.189 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.192+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:03.332 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.336+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:34:03.386 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.389+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:03.403 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.407+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:03.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.418+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:03.453 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.457+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:34:03.520 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.524+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:03.546 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.550+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:03.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:03.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1150', time=2019-12-09T23:21:05.703061+0000, chain.objs.size()=9 2019-12-09T21:34:03.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.72__multipart_key1.2~WofZynNmnSUWJlh0ntIsHw3gaOje3z5.1 2019-12-09T21:34:03.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.72__shadow_key1.2~WofZynNmnSUWJlh0ntIsHw3gaOje3z5.1_1 2019-12-09T21:34:03.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.72__multipart_key1.2~WofZynNmnSUWJlh0ntIsHw3gaOje3z5.2 2019-12-09T21:34:03.567 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.72__shadow_key1.2~WofZynNmnSUWJlh0ntIsHw3gaOje3z5.2_1 2019-12-09T21:34:03.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.72__multipart_key1.2~WofZynNmnSUWJlh0ntIsHw3gaOje3z5.3 2019-12-09T21:34:03.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.72__shadow_key1.2~WofZynNmnSUWJlh0ntIsHw3gaOje3z5.3_1 2019-12-09T21:34:03.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.72__multipart_key1.2~WofZynNmnSUWJlh0ntIsHw3gaOje3z5.4 2019-12-09T21:34:03.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.72__shadow_key1.2~WofZynNmnSUWJlh0ntIsHw3gaOje3z5.4_1 2019-12-09T21:34:03.568 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:03.570+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.72__multipart_key1.2~WofZynNmnSUWJlh0ntIsHw3gaOje3z5.5 2019-12-09T21:34:04.011 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.015+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:04.136 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.140+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:34:04.183 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.186+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.208+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.229+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.229+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1052', time=2019-12-09T23:19:56.752647+0000, chain.objs.size()=5 2019-12-09T21:34:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.229+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.66__shadow_.UxImiWvaQBBphVOZvQe_2BbyD9I645W_1 2019-12-09T21:34:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.229+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.66__shadow_.UxImiWvaQBBphVOZvQe_2BbyD9I645W_2 2019-12-09T21:34:04.226 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.229+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.66__shadow_.UxImiWvaQBBphVOZvQe_2BbyD9I645W_3 2019-12-09T21:34:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.229+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.66__shadow_.UxImiWvaQBBphVOZvQe_2BbyD9I645W_4 2019-12-09T21:34:04.227 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.229+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.66__shadow_.UxImiWvaQBBphVOZvQe_2BbyD9I645W_5 2019-12-09T21:34:04.481 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.485+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.582+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:34:04.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.625+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.643+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:04.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.652+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:04.708 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.712+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:34:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.758+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.776+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=2, truncated=0, next_marker='' 2019-12-09T21:34:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1076', time=2019-12-09T23:19:00.141463+0000, chain.objs.size()=9 2019-12-09T21:34:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~Ewkm8gkQ4UbHFxjnVwGFFkkw9iBfXO6.1 2019-12-09T21:34:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~Ewkm8gkQ4UbHFxjnVwGFFkkw9iBfXO6.1_1 2019-12-09T21:34:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~Ewkm8gkQ4UbHFxjnVwGFFkkw9iBfXO6.2 2019-12-09T21:34:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~Ewkm8gkQ4UbHFxjnVwGFFkkw9iBfXO6.2_1 2019-12-09T21:34:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~Ewkm8gkQ4UbHFxjnVwGFFkkw9iBfXO6.3 2019-12-09T21:34:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~Ewkm8gkQ4UbHFxjnVwGFFkkw9iBfXO6.3_1 2019-12-09T21:34:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~Ewkm8gkQ4UbHFxjnVwGFFkkw9iBfXO6.4 2019-12-09T21:34:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~Ewkm8gkQ4UbHFxjnVwGFFkkw9iBfXO6.4_1 2019-12-09T21:34:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~Ewkm8gkQ4UbHFxjnVwGFFkkw9iBfXO6.5 2019-12-09T21:34:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.796+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1182', time=2019-12-09T23:21:22.929944+0000, chain.objs.size()=9 2019-12-09T21:34:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.797+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file.mpg.2~0rwElgKtxSo86XHYmnBbtjJLuSodTtI.1 2019-12-09T21:34:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.797+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__shadow_file.mpg.2~0rwElgKtxSo86XHYmnBbtjJLuSodTtI.1_1 2019-12-09T21:34:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:04.797+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file.mpg.2~0rwElgKtxSo86XHYmnBbtjJLuSodTtI.2 2019-12-09T21:34:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.014+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__shadow_file.mpg.2~0rwElgKtxSo86XHYmnBbtjJLuSodTtI.2_1 2019-12-09T21:34:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.083+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file.mpg.2~0rwElgKtxSo86XHYmnBbtjJLuSodTtI.3 2019-12-09T21:34:05.222 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.226+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__shadow_file.mpg.2~0rwElgKtxSo86XHYmnBbtjJLuSodTtI.3_1 2019-12-09T21:34:05.222 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.226+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file.mpg.2~0rwElgKtxSo86XHYmnBbtjJLuSodTtI.4 2019-12-09T21:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.226+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__shadow_file.mpg.2~0rwElgKtxSo86XHYmnBbtjJLuSodTtI.4_1 2019-12-09T21:34:05.223 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.226+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.73__multipart_file.mpg.2~0rwElgKtxSo86XHYmnBbtjJLuSodTtI.5 2019-12-09T21:34:05.570 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.574+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:05.695 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.698+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:34:05.737 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.741+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:05.752 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.756+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.768+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:05.765 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.768+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1088', time=2019-12-09T23:19:53.176710+0000, chain.objs.size()=7 2019-12-09T21:34:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.768+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~n3ceeaogf7cqLVRoRm9I7GfWirIJ2vp.1 2019-12-09T21:34:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.769+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~n3ceeaogf7cqLVRoRm9I7GfWirIJ2vp.1_1 2019-12-09T21:34:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.769+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~n3ceeaogf7cqLVRoRm9I7GfWirIJ2vp.1_2 2019-12-09T21:34:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.769+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~n3ceeaogf7cqLVRoRm9I7GfWirIJ2vp.2 2019-12-09T21:34:05.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.769+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~n3ceeaogf7cqLVRoRm9I7GfWirIJ2vp.2_1 2019-12-09T21:34:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.769+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~n3ceeaogf7cqLVRoRm9I7GfWirIJ2vp.2_2 2019-12-09T21:34:05.767 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:05.769+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~n3ceeaogf7cqLVRoRm9I7GfWirIJ2vp.3 2019-12-09T21:34:06.038 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.041+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:06.217 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.220+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:34:06.269 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.273+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:06.291 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.294+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:06.312 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.315+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=3, truncated=0, next_marker='' 2019-12-09T21:34:06.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.315+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1039', time=2019-12-09T23:16:15.625700+0000, chain.objs.size()=9 2019-12-09T21:34:06.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.315+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.1 2019-12-09T21:34:06.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.315+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__shadow_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.1_1 2019-12-09T21:34:06.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.315+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.2 2019-12-09T21:34:06.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.315+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__shadow_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.2_1 2019-12-09T21:34:06.313 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.316+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.3 2019-12-09T21:34:06.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.316+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__shadow_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.3_1 2019-12-09T21:34:06.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.316+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.4 2019-12-09T21:34:06.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.316+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__shadow_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.4_1 2019-12-09T21:34:06.314 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.316+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.5 2019-12-09T21:34:06.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.316+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1037', time=2019-12-09T23:16:20.233723+0000, chain.objs.size()=9 2019-12-09T21:34:06.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.316+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.1 2019-12-09T21:34:06.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.316+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__shadow_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.1_1 2019-12-09T21:34:06.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.316+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.2 2019-12-09T21:34:06.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.637+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__shadow_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.2_1 2019-12-09T21:34:06.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.638+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.3 2019-12-09T21:34:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.690+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__shadow_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.3_1 2019-12-09T21:34:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.690+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.4 2019-12-09T21:34:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.690+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__shadow_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.4_1 2019-12-09T21:34:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.690+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.64__multipart_key1.2~39mKbITV4uyHUWQJTOhRfXDHZIO_N-2.5 2019-12-09T21:34:06.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.690+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1257', time=2019-12-09T23:24:29.929671+0000, chain.objs.size()=9 2019-12-09T21:34:06.688 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.690+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.76__multipart_key1.2~QVrzbS4Cm3Z4B2ZD2IET9xZlhNlN4L5.1 2019-12-09T21:34:06.688 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.690+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.76__shadow_key1.2~QVrzbS4Cm3Z4B2ZD2IET9xZlhNlN4L5.1_1 2019-12-09T21:34:06.688 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.690+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.76__multipart_key1.2~QVrzbS4Cm3Z4B2ZD2IET9xZlhNlN4L5.2 2019-12-09T21:34:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.879+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.76__shadow_key1.2~QVrzbS4Cm3Z4B2ZD2IET9xZlhNlN4L5.2_1 2019-12-09T21:34:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.879+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.76__multipart_key1.2~QVrzbS4Cm3Z4B2ZD2IET9xZlhNlN4L5.3 2019-12-09T21:34:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.999+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.76__shadow_key1.2~QVrzbS4Cm3Z4B2ZD2IET9xZlhNlN4L5.3_1 2019-12-09T21:34:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.999+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.76__multipart_key1.2~QVrzbS4Cm3Z4B2ZD2IET9xZlhNlN4L5.4 2019-12-09T21:34:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.999+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.76__shadow_key1.2~QVrzbS4Cm3Z4B2ZD2IET9xZlhNlN4L5.4_1 2019-12-09T21:34:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:06.999+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.76__multipart_key1.2~QVrzbS4Cm3Z4B2ZD2IET9xZlhNlN4L5.5 2019-12-09T21:34:07.223 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.226+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:07.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.436+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:34:07.497 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.501+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:07.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.523+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:07.537 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:07.538 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1908', time=2019-12-09T23:33:49.590942+0000, chain.objs.size()=9 2019-12-09T21:34:07.538 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.119__multipart_key1/data/file.mpg.2~Z4ZLtnA92MfuQP6nguLbGna7ISxCrmv.1 2019-12-09T21:34:07.538 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.119__shadow_key1/data/file.mpg.2~Z4ZLtnA92MfuQP6nguLbGna7ISxCrmv.1_1 2019-12-09T21:34:07.538 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.119__multipart_key1/data/file.mpg.2~Z4ZLtnA92MfuQP6nguLbGna7ISxCrmv.2 2019-12-09T21:34:07.538 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.119__shadow_key1/data/file.mpg.2~Z4ZLtnA92MfuQP6nguLbGna7ISxCrmv.2_1 2019-12-09T21:34:07.538 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.119__multipart_key1/data/file.mpg.2~Z4ZLtnA92MfuQP6nguLbGna7ISxCrmv.3 2019-12-09T21:34:07.538 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.119__shadow_key1/data/file.mpg.2~Z4ZLtnA92MfuQP6nguLbGna7ISxCrmv.3_1 2019-12-09T21:34:07.539 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.119__multipart_key1/data/file.mpg.2~Z4ZLtnA92MfuQP6nguLbGna7ISxCrmv.4 2019-12-09T21:34:07.539 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.119__shadow_key1/data/file.mpg.2~Z4ZLtnA92MfuQP6nguLbGna7ISxCrmv.4_1 2019-12-09T21:34:07.539 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.541+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.119__multipart_key1/data/file.mpg.2~Z4ZLtnA92MfuQP6nguLbGna7ISxCrmv.5 2019-12-09T21:34:07.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.786+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:07.920 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.924+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:34:07.966 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.970+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:07.986 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:07.989+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:08.003 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1060', time=2019-12-09T23:17:45.078197+0000, chain.objs.size()=9 2019-12-09T21:34:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VXU30Lt-I-Ukpm1-85EXXdPngqq-8R5.1 2019-12-09T21:34:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~VXU30Lt-I-Ukpm1-85EXXdPngqq-8R5.1_1 2019-12-09T21:34:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VXU30Lt-I-Ukpm1-85EXXdPngqq-8R5.2 2019-12-09T21:34:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~VXU30Lt-I-Ukpm1-85EXXdPngqq-8R5.2_1 2019-12-09T21:34:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VXU30Lt-I-Ukpm1-85EXXdPngqq-8R5.3 2019-12-09T21:34:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~VXU30Lt-I-Ukpm1-85EXXdPngqq-8R5.3_1 2019-12-09T21:34:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VXU30Lt-I-Ukpm1-85EXXdPngqq-8R5.4 2019-12-09T21:34:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~VXU30Lt-I-Ukpm1-85EXXdPngqq-8R5.4_1 2019-12-09T21:34:08.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.006+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~VXU30Lt-I-Ukpm1-85EXXdPngqq-8R5.5 2019-12-09T21:34:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.376+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:08.463 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.466+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:34:08.520 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.524+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:08.542 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.545+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:08.552 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.555+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:08.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.614+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:34:08.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.674+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:08.701 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.700+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2019-12-09T21:34:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5', time=2019-12-09T23:14:16.590885+0000, chain.objs.size()=9 2019-12-09T21:34:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.52__multipart_key1.2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5.1 2019-12-09T21:34:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.52__shadow_key1.2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5.1_1 2019-12-09T21:34:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.52__multipart_key1.2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5.2 2019-12-09T21:34:08.729 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.52__shadow_key1.2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5.2_1 2019-12-09T21:34:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.52__multipart_key1.2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5.3 2019-12-09T21:34:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.52__shadow_key1.2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5.3_1 2019-12-09T21:34:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.52__multipart_key1.2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5.4 2019-12-09T21:34:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.52__shadow_key1.2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5.4_1 2019-12-09T21:34:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.52__multipart_key1.2~Ky9ubFUxPSOkKTuCJQ7t_PR7hOX-vJ5.5 2019-12-09T21:34:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1082', time=2019-12-09T23:19:27.337258+0000, chain.objs.size()=7 2019-12-09T21:34:08.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~FA_PEQUAsbSVuhNlSl0fVRNaHw65d0B.1 2019-12-09T21:34:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~FA_PEQUAsbSVuhNlSl0fVRNaHw65d0B.1_1 2019-12-09T21:34:08.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:08.732+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~FA_PEQUAsbSVuhNlSl0fVRNaHw65d0B.1_2 2019-12-09T21:34:09.015 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.018+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~FA_PEQUAsbSVuhNlSl0fVRNaHw65d0B.2 2019-12-09T21:34:09.015 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.019+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~FA_PEQUAsbSVuhNlSl0fVRNaHw65d0B.2_1 2019-12-09T21:34:09.015 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.019+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~FA_PEQUAsbSVuhNlSl0fVRNaHw65d0B.2_2 2019-12-09T21:34:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.130+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~FA_PEQUAsbSVuhNlSl0fVRNaHw65d0B.3 2019-12-09T21:34:09.126 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.130+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1131', time=2019-12-09T23:20:43.154102+0000, chain.objs.size()=9 2019-12-09T21:34:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.130+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.71__multipart_key1.2~1lYgN4Px5sek0ONV-bzmcsrcyLcllMa.1 2019-12-09T21:34:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.130+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.71__shadow_key1.2~1lYgN4Px5sek0ONV-bzmcsrcyLcllMa.1_1 2019-12-09T21:34:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.130+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.71__multipart_key1.2~1lYgN4Px5sek0ONV-bzmcsrcyLcllMa.2 2019-12-09T21:34:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.130+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.71__shadow_key1.2~1lYgN4Px5sek0ONV-bzmcsrcyLcllMa.2_1 2019-12-09T21:34:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.130+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.71__multipart_key1.2~1lYgN4Px5sek0ONV-bzmcsrcyLcllMa.3 2019-12-09T21:34:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.130+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.71__shadow_key1.2~1lYgN4Px5sek0ONV-bzmcsrcyLcllMa.3_1 2019-12-09T21:34:09.127 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.130+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.71__multipart_key1.2~1lYgN4Px5sek0ONV-bzmcsrcyLcllMa.4 2019-12-09T21:34:09.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.370+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.71__shadow_key1.2~1lYgN4Px5sek0ONV-bzmcsrcyLcllMa.4_1 2019-12-09T21:34:09.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.370+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.71__multipart_key1.2~1lYgN4Px5sek0ONV-bzmcsrcyLcllMa.5 2019-12-09T21:34:09.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.370+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1892', time=2019-12-09T23:33:33.392613+0000, chain.objs.size()=9 2019-12-09T21:34:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.370+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.118__multipart_key1.2~C6A2K_V3u9cXx-9tkW9_Ots37mwZoH9.1 2019-12-09T21:34:09.367 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.370+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.118__shadow_key1.2~C6A2K_V3u9cXx-9tkW9_Ots37mwZoH9.1_1 2019-12-09T21:34:09.427 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.430+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.118__multipart_key1.2~C6A2K_V3u9cXx-9tkW9_Ots37mwZoH9.2 2019-12-09T21:34:09.471 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.474+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.118__shadow_key1.2~C6A2K_V3u9cXx-9tkW9_Ots37mwZoH9.2_1 2019-12-09T21:34:09.471 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.474+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.118__multipart_key1.2~C6A2K_V3u9cXx-9tkW9_Ots37mwZoH9.3 2019-12-09T21:34:09.471 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.474+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.118__shadow_key1.2~C6A2K_V3u9cXx-9tkW9_Ots37mwZoH9.3_1 2019-12-09T21:34:09.497 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.501+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.118__multipart_key1.2~C6A2K_V3u9cXx-9tkW9_Ots37mwZoH9.4 2019-12-09T21:34:09.497 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.501+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.118__shadow_key1.2~C6A2K_V3u9cXx-9tkW9_Ots37mwZoH9.4_1 2019-12-09T21:34:09.498 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.501+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.118__multipart_key1.2~C6A2K_V3u9cXx-9tkW9_Ots37mwZoH9.5 2019-12-09T21:34:09.839 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.843+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:09.952 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:09.955+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:34:09.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.002+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:10.017 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.021+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:10.030 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.033+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:10.030 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.033+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='1ad16726-ce4c-4982-bd45-b63e0fe58407.4557.1094', time=2019-12-09T23:20:00.222832+0000, chain.objs.size()=7 2019-12-09T21:34:10.030 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.033+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~rW9NphJqC1GxIN7BpyhgX-HuYrwQQRV.1 2019-12-09T21:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.033+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~rW9NphJqC1GxIN7BpyhgX-HuYrwQQRV.1_1 2019-12-09T21:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.033+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~rW9NphJqC1GxIN7BpyhgX-HuYrwQQRV.1_2 2019-12-09T21:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.033+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~rW9NphJqC1GxIN7BpyhgX-HuYrwQQRV.2 2019-12-09T21:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.033+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~rW9NphJqC1GxIN7BpyhgX-HuYrwQQRV.2_1 2019-12-09T21:34:10.031 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.033+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__shadow_key1.2~rW9NphJqC1GxIN7BpyhgX-HuYrwQQRV.2_2 2019-12-09T21:34:10.032 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.033+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.67__multipart_key1.2~rW9NphJqC1GxIN7BpyhgX-HuYrwQQRV.3 2019-12-09T21:34:10.340 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.344+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:10.459 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.463+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:34:10.516 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.520+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:10.538 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.541+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:10.549 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.553+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:10.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.613+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:34:10.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.659+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:10.679 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.676+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:10.680 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.684+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:10.722 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.725+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:34:10.792 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.795+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:10.821 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.825+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:10.831 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.835+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:10.880 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.883+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:34:10.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.965+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:10.994 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:10.997+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:11.008 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.012+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:11.070 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.074+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:34:11.132 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.135+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:11.150 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.153+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:11.174 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:11.174 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv', time=2019-12-09T23:25:23.105274+0000, chain.objs.size()=37 2019-12-09T21:34:11.174 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.1 2019-12-09T21:34:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.2 2019-12-09T21:34:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.3 2019-12-09T21:34:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.4 2019-12-09T21:34:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.5 2019-12-09T21:34:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.6 2019-12-09T21:34:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.7 2019-12-09T21:34:11.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.8 2019-12-09T21:34:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.9 2019-12-09T21:34:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.10 2019-12-09T21:34:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.11 2019-12-09T21:34:11.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.177+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.12 2019-12-09T21:34:11.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.407+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.13 2019-12-09T21:34:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.508+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.14 2019-12-09T21:34:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.508+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.15 2019-12-09T21:34:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.508+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.H3B7DMZf65f3K1UQq_dUNgvdwyDXXxq.16 2019-12-09T21:34:11.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.508+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.17 2019-12-09T21:34:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.552+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.18 2019-12-09T21:34:11.549 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.552+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.C73pprw8ZwSAA9LsSGHOOcHmQv8WeFB.19 2019-12-09T21:34:11.563 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.566+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.20 2019-12-09T21:34:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.629+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.21 2019-12-09T21:34:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.629+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.22 2019-12-09T21:34:11.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.629+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.23 2019-12-09T21:34:11.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.629+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.24 2019-12-09T21:34:11.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.776+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.25 2019-12-09T21:34:11.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.26 2019-12-09T21:34:11.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.27 2019-12-09T21:34:11.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.28 2019-12-09T21:34:11.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.29 2019-12-09T21:34:11.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.30 2019-12-09T21:34:11.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.31 2019-12-09T21:34:11.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.32 2019-12-09T21:34:11.879 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.33 2019-12-09T21:34:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.34 2019-12-09T21:34:11.880 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:11.881+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.35 2019-12-09T21:34:12.075 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.079+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.36 2019-12-09T21:34:12.125 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.129+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.78__multipart_key1.2~bYXrBlEWoP6w27qtz_LUCCB7Yah-wqv.37 2019-12-09T21:34:12.237 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.238+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:12.336 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.340+0000 7f059a35aec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:34:12.411 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.415+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:12.441 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.444+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:12.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.457+0000 7f059a35aec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=1, truncated=0, next_marker='' 2019-12-09T21:34:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.457+0000 7f059a35aec0 20 garbage collection: RGWGC::process iterating over entry tag='2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn', time=2019-12-09T23:21:58.884919+0000, chain.objs.size()=9 2019-12-09T21:34:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.457+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.74__multipart_key1.2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn.7 2019-12-09T21:34:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.458+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.74__shadow_key1.2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn.7_1 2019-12-09T21:34:12.456 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.458+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.74__multipart_key1.2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn.8 2019-12-09T21:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.458+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.74__shadow_key1.2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn.8_1 2019-12-09T21:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.458+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.74__multipart_key1.2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn.9 2019-12-09T21:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.458+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.74__shadow_key1.2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn.9_1 2019-12-09T21:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.458+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.74__multipart_key1.2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn.10 2019-12-09T21:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.458+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.74__shadow_key1.2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn.10_1 2019-12-09T21:34:12.457 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.458+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.74__multipart_key1.2~WMABh2kv6btqDc3aiq6bUqMLS4QBcDn.11 2019-12-09T21:34:12.791 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.794+0000 7f059a35aec0 5 garbage collection: RGWGC::process removing entries, marker: 2019-12-09T21:34:12.917 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.921+0000 7f059a35aec0 20 remove_watcher() i=0 2019-12-09T21:34:12.917 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.921+0000 7f059a35aec0 2 removed watcher, disabling cache 2019-12-09T21:34:12.956 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.960+0000 7f059a35aec0 20 remove_watcher() i=1 2019-12-09T21:34:12.988 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:12.991+0000 7f059a35aec0 20 remove_watcher() i=2 2019-12-09T21:34:13.048 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.052+0000 7f059a35aec0 20 remove_watcher() i=3 2019-12-09T21:34:13.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.100+0000 7f059a35aec0 20 remove_watcher() i=4 2019-12-09T21:34:13.132 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.136+0000 7f059a35aec0 20 remove_watcher() i=5 2019-12-09T21:34:13.165 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.168+0000 7f059a35aec0 20 remove_watcher() i=6 2019-12-09T21:34:13.211 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.215+0000 7f059a35aec0 20 remove_watcher() i=7 2019-12-09T21:34:13.220 INFO:teuthology.orchestra.run.smithi067:> radosgw-admin gc process --include-all 2019-12-09T21:34:13.358 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.361+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.383 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.386+0000 7f67cda35ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:13.383 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.386+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.392 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.396+0000 7f67cda35ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:13.392 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.396+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.400+0000 7f67cda35ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:13.397 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.400+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.408+0000 7f67cda35ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.597+0000 7f67cda35ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:34:13.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.597+0000 7f67cda35ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:34:13.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.597+0000 7f67cda35ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:34:13.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.597+0000 7f67cda35ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:34:13.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.597+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.604+0000 7f67cda35ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:13.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.604+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.614+0000 7f67cda35ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:13.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.614+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.621+0000 7f67cda35ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:13.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.621+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.636+0000 7f67cda35ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:13.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.636+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.641+0000 7f67cda35ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:13.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.641+0000 7f67cda35ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:34:13.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.641+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.646+0000 7f67cda35ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:13.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.646+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.654+0000 7f67cda35ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:13.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.654+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.667+0000 7f67cda35ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:13.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.667+0000 7f67cda35ec0 20 zonegroup default 2019-12-09T21:34:13.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.667+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.672+0000 7f67cda35ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.672+0000 7f67cda35ec0 10 Cannot find current period zone using local zone 2019-12-09T21:34:13.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.672+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.680+0000 7f67cda35ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:13.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.680+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.694+0000 7f67cda35ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:13.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.694+0000 7f67cda35ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.704+0000 7f67cda35ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.705+0000 7f67cda35ec0 20 zone default found 2019-12-09T21:34:13.701 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.705+0000 7f67cda35ec0 20 started sync module instance, tier type = 2019-12-09T21:34:13.702 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.705+0000 7f67cda35ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:34:13.950 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:13.953+0000 7f67bac5f700 0 -- 172.21.15.67:0/3470830320 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7f67b40016c0 msgr2=0x7f67b40067d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:34:14.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.105+0000 7f67cda35ec0 20 add_watcher() i=0 2019-12-09T21:34:14.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.105+0000 7f67cda35ec0 20 add_watcher() i=1 2019-12-09T21:34:14.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.105+0000 7f67cda35ec0 20 add_watcher() i=2 2019-12-09T21:34:14.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.105+0000 7f67cda35ec0 20 add_watcher() i=3 2019-12-09T21:34:14.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.105+0000 7f67cda35ec0 20 add_watcher() i=4 2019-12-09T21:34:14.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.105+0000 7f67cda35ec0 20 add_watcher() i=5 2019-12-09T21:34:14.106 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.109+0000 7f67cda35ec0 20 add_watcher() i=6 2019-12-09T21:34:14.140 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.143+0000 7f67cda35ec0 20 add_watcher() i=7 2019-12-09T21:34:14.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.143+0000 7f67cda35ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:34:14.141 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.144+0000 7f678b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:34:14.145 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:14.149+0000 7f67bb460700 0 -- 172.21.15.67:0/3638392104 >> v1:172.21.15.67:6789/0 conn(0x55ac7f2e99d0 legacy=0x55ac7f2e8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-09T21:34:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.446+0000 7f66f9ffb700 20 reqs_thread_entry: start 2019-12-09T21:34:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.446+0000 7f66f37fe700 20 reqs_thread_entry: start 2019-12-09T21:34:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.446+0000 7f67cda35ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:34:15.443 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.447+0000 7f66f1ffb700 20 reqs_thread_entry: start 2019-12-09T21:34:15.444 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.447+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2019-12-09T21:34:15.492 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.495+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:15.507 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.511+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:15.516 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.520+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:15.553 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.556+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2019-12-09T21:34:15.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.610+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:15.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.631+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:15.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.640+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:15.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.666+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2019-12-09T21:34:15.717 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.713+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:15.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.733+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:15.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.743+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:15.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.777+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2019-12-09T21:34:15.825 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.828+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:15.844 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.847+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:15.856 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.860+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:15.887 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.891+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2019-12-09T21:34:15.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.968+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:15.994 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:15.997+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:16.009 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.012+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.058 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.062+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2019-12-09T21:34:16.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.126+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.149 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.152+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:16.160 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.163+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.218 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.222+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2019-12-09T21:34:16.297 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.300+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.320 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.323+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:16.332 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.335+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.386 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.389+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2019-12-09T21:34:16.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.464+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.485 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.489+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.500+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.539 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.542+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2019-12-09T21:34:16.581 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.584+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.626+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:16.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.637+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.684 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.687+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2019-12-09T21:34:16.755 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.759+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.776 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.779+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:16.787 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.790+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.831 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.835+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2019-12-09T21:34:16.906 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.910+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.922 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.926+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:16.932 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.935+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:16.980 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:16.984+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2019-12-09T21:34:17.051 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.055+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.077+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:17.085 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.088+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.136 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.139+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2019-12-09T21:34:17.179 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.182+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.198+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.206+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.239+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2019-12-09T21:34:17.276 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.278+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.290 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.293+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:17.299 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.303+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.326 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.330+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2019-12-09T21:34:17.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.387+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.399 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.403+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:17.408 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.411+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.455 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.457+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2019-12-09T21:34:17.528 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.531+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.566 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.569+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:17.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.591+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.646+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2019-12-09T21:34:17.701 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.704+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.721 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.724+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:17.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.733+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.778 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.781+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2019-12-09T21:34:17.860 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.863+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.890+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:17.898 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.902+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:17.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:17.972+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2019-12-09T21:34:18.030 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.033+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.050 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.054+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:18.064 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.068+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.133 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.136+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2019-12-09T21:34:18.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.179+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.197 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.200+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:18.207 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.210+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.239 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.243+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2019-12-09T21:34:18.317 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.321+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.343 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.346+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:18.356 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.359+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.421+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2019-12-09T21:34:18.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.469+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.485 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.487+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:18.514 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.517+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.562 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.566+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2019-12-09T21:34:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.631+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.658+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:18.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.667+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.706 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.709+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2019-12-09T21:34:18.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.778+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.805 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.803+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:18.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.813+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.870 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.873+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2019-12-09T21:34:18.940 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.943+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:18.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.966+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:18.978 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:18.981+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.018 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.021+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2019-12-09T21:34:19.083 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.086+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.105+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:19.113 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.116+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.162+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2019-12-09T21:34:19.241 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.244+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.268 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.272+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:19.286 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.289+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.347+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2019-12-09T21:34:19.441 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.444+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.462 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.465+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.481+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.519 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.522+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2019-12-09T21:34:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.581+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.609+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:19.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.623+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.673+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2019-12-09T21:34:19.733 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.737+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.763 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.765+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:19.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.775+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.808 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.811+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2019-12-09T21:34:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.867+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.894+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:19.905 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.908+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:19.961 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:19.964+0000 7f67cda35ec0 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2019-12-09T21:34:20.017 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.010+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:20.029 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.031+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_list returned ENOENT for non expired entries, so setting cache entry to TRUE 2019-12-09T21:34:20.042 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.044+0000 7f67cda35ec0 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2019-12-09T21:34:20.130 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.133+0000 7f67cda35ec0 20 remove_watcher() i=0 2019-12-09T21:34:20.130 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.133+0000 7f67cda35ec0 2 removed watcher, disabling cache 2019-12-09T21:34:20.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.178+0000 7f67cda35ec0 20 remove_watcher() i=1 2019-12-09T21:34:20.217 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.221+0000 7f67cda35ec0 20 remove_watcher() i=2 2019-12-09T21:34:20.291 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.295+0000 7f67cda35ec0 20 remove_watcher() i=3 2019-12-09T21:34:20.360 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.364+0000 7f67cda35ec0 20 remove_watcher() i=4 2019-12-09T21:34:20.424 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.428+0000 7f67cda35ec0 20 remove_watcher() i=5 2019-12-09T21:34:20.472 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.476+0000 7f67cda35ec0 20 remove_watcher() i=6 2019-12-09T21:34:20.530 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:20.534+0000 7f67cda35ec0 20 remove_watcher() i=7 2019-12-09T21:34:20.541 INFO:teuthology.run_tasks:Running task s3tests... 2019-12-09T21:34:20.571 DEBUG:tasks.s3tests:s3tests config is {'client.0': {'rgw_server': 'client.0', 'force-branch': 'public-access-qa', 'git_remote': 'git://github.com/theanalyst/', 'slow_backend': True}} 2019-12-09T21:34:20.571 INFO:tasks.s3tests:Downloading s3-tests... 2019-12-09T21:34:20.571 INFO:tasks.s3tests:Using branch 'public-access-qa' for s3tests 2019-12-09T21:34:20.571 INFO:teuthology.orchestra.run.smithi067:> git clone -b public-access-qa git://github.com/theanalyst/s3-tests.git /home/ubuntu/cephtest/s3-tests 2019-12-09T21:34:20.743 INFO:teuthology.orchestra.run.smithi067.stdout:Cloning into '/home/ubuntu/cephtest/s3-tests'... 2019-12-09T21:34:21.495 INFO:tasks.s3tests:Creating rgw users... 2019-12-09T21:34:21.496 DEBUG:tasks.s3tests:Creating user bar.client.0 on client.0 2019-12-09T21:34:21.497 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid bar.client.0 --display-name 'Mr. bar.client.0' --access-key ODWNEOWYVZRFIQZWQXXL --secret +MBXq46cTDvGI2vAaXv8QCY2g+3lAUcnYMvH6LwcX+Ue015CUWZ+FA== --email bar.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2019-12-09T21:34:21.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.635+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.668+0000 7f70d0754ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:21.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.668+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.676+0000 7f70d0754ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:21.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.676+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.678 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.681+0000 7f70d0754ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:21.678 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.681+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.685 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.689+0000 7f70d0754ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:21.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.855+0000 7f70d0754ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:34:21.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.855+0000 7f70d0754ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:34:21.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.855+0000 7f70d0754ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:34:21.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.855+0000 7f70d0754ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:34:21.852 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.855+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.859 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.863+0000 7f70d0754ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:21.859 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.863+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.875+0000 7f70d0754ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:21.872 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.875+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.880 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.884+0000 7f70d0754ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:21.880 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.884+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.896+0000 7f70d0754ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:21.893 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.896+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.906+0000 7f70d0754ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.906+0000 7f70d0754ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:34:21.903 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.906+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.911+0000 7f70d0754ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:21.908 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.911+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.930+0000 7f70d0754ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:21.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.930+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.936 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.939+0000 7f70d0754ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.939+0000 7f70d0754ec0 20 zonegroup default 2019-12-09T21:34:21.937 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.939+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.945+0000 7f70d0754ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.945+0000 7f70d0754ec0 10 Cannot find current period zone using local zone 2019-12-09T21:34:21.942 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.945+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.949+0000 7f70d0754ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:21.946 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.949+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.960+0000 7f70d0754ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:21.956 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.960+0000 7f70d0754ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.968+0000 7f70d0754ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.968+0000 7f70d0754ec0 20 zone default found 2019-12-09T21:34:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.968+0000 7f70d0754ec0 20 started sync module instance, tier type = 2019-12-09T21:34:21.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:21.968+0000 7f70d0754ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:34:22.383 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.386+0000 7f70d0754ec0 20 add_watcher() i=0 2019-12-09T21:34:22.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.386+0000 7f70d0754ec0 20 add_watcher() i=1 2019-12-09T21:34:22.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.386+0000 7f70d0754ec0 20 add_watcher() i=2 2019-12-09T21:34:22.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.386+0000 7f70d0754ec0 20 add_watcher() i=3 2019-12-09T21:34:22.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.386+0000 7f70d0754ec0 20 add_watcher() i=4 2019-12-09T21:34:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.386+0000 7f70d0754ec0 20 add_watcher() i=5 2019-12-09T21:34:22.385 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.386+0000 7f70d0754ec0 20 add_watcher() i=6 2019-12-09T21:34:22.429 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.432+0000 7f70d0754ec0 20 add_watcher() i=7 2019-12-09T21:34:22.429 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.432+0000 7f70d0754ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:34:22.429 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.432+0000 7f708e7dc700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:34:22.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.436+0000 7f70bd97e700 0 -- 172.21.15.67:0/3055756300 >> v1:172.21.15.67:6789/0 conn(0x56071f5b7ae0 legacy=0x56071f5b9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-09T21:34:22.474 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:22.477+0000 7f70b7fff700 0 -- 172.21.15.67:0/1215819420 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x56071f5ccb60 msgr2=0x56071f5cef80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:34:23.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.973+0000 7f7004ff9700 20 reqs_thread_entry: start 2019-12-09T21:34:23.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.974+0000 7f6ffe7fc700 20 reqs_thread_entry: start 2019-12-09T21:34:23.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.974+0000 7f70d0754ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:34:23.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.974+0000 7f6ffcff9700 20 reqs_thread_entry: start 2019-12-09T21:34:23.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.974+0000 7f70d0754ec0 20 get_system_obj_state: rctx=0x7ffe30d9b438 obj=default.rgw.meta:users.uid:bar.client.0 state=0x56071f639620 s->prefetch_data=0 2019-12-09T21:34:23.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.974+0000 7f70d0754ec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2019-12-09T21:34:23.983 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.986+0000 7f70d0754ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2019-12-09T21:34:23.983 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.986+0000 7f70d0754ec0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-09T21:34:23.983 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.986+0000 7f70d0754ec0 20 get_system_obj_state: rctx=0x7ffe30d9b448 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x56071f63b3b0 s->prefetch_data=0 2019-12-09T21:34:23.983 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.986+0000 7f70d0754ec0 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2019-12-09T21:34:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.993+0000 7f70d0754ec0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2019-12-09T21:34:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.993+0000 7f70d0754ec0 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-09T21:34:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.993+0000 7f70d0754ec0 20 get_system_obj_state: rctx=0x7ffe30d9b448 obj=default.rgw.meta:users.keys:ODWNEOWYVZRFIQZWQXXL state=0x56071f63b3b0 s->prefetch_data=0 2019-12-09T21:34:23.989 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:23.993+0000 7f70d0754ec0 10 cache get: name=default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL : miss 2019-12-09T21:34:23.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.002+0000 7f70d0754ec0 10 cache put: name=default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL info.flags=0x0 2019-12-09T21:34:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.002+0000 7f70d0754ec0 10 adding default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL to cache LRU end 2019-12-09T21:34:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.002+0000 7f70d0754ec0 20 get_system_obj_state: rctx=0x7ffe30d9b058 obj=default.rgw.meta:users.keys:ODWNEOWYVZRFIQZWQXXL state=0x56071f63b3b0 s->prefetch_data=0 2019-12-09T21:34:24.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.002+0000 7f70d0754ec0 10 cache get: name=default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL : type miss (requested=0x6, cached=0x0) 2019-12-09T21:34:24.012 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.010+0000 7f70d0754ec0 10 cache put: name=default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL info.flags=0x0 2019-12-09T21:34:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.010+0000 7f70d0754ec0 10 moving default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL to cache LRU end 2019-12-09T21:34:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.010+0000 7f70d0754ec0 20 get_system_obj_state: rctx=0x7ffe30d9b1d8 obj=default.rgw.meta:users.keys:ODWNEOWYVZRFIQZWQXXL state=0x56071f63b3b0 s->prefetch_data=0 2019-12-09T21:34:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.010+0000 7f70d0754ec0 10 cache get: name=default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL : type miss (requested=0x6, cached=0x0) 2019-12-09T21:34:24.014 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.018+0000 7f70d0754ec0 10 cache put: name=default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL info.flags=0x0 2019-12-09T21:34:24.015 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.018+0000 7f70d0754ec0 10 moving default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL to cache LRU end 2019-12-09T21:34:24.062 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.065+0000 7f70d0754ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-09T21:34:24.062 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.065+0000 7f70d0754ec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-09T21:34:24.062 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.065+0000 7f70d0754ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=636 2019-12-09T21:34:24.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.076+0000 7f709effd700 10 RGWWatcher::handle_notify() notify_id 592705486877 cookie 94588590743952 notifier 4965 bl.length()=636 2019-12-09T21:34:24.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.076+0000 7f709effd700 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-09T21:34:24.073 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.077+0000 7f709effd700 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-09T21:34:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.303+0000 7f70d0754ec0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2019-12-09T21:34:24.300 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.303+0000 7f70d0754ec0 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-09T21:34:24.301 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.303+0000 7f70d0754ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=194 2019-12-09T21:34:24.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.312+0000 7f709effd700 10 RGWWatcher::handle_notify() notify_id 592705486878 cookie 94588590743952 notifier 4965 bl.length()=194 2019-12-09T21:34:24.309 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.312+0000 7f709effd700 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2019-12-09T21:34:24.310 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.312+0000 7f709effd700 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-09T21:34:24.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.941+0000 7f70d0754ec0 10 cache put: name=default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL info.flags=0x7 2019-12-09T21:34:24.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.941+0000 7f70d0754ec0 10 moving default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL to cache LRU end 2019-12-09T21:34:24.938 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.941+0000 7f70d0754ec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=186 2019-12-09T21:34:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.958+0000 7f709effd700 10 RGWWatcher::handle_notify() notify_id 592705486882 cookie 94588590749808 notifier 4965 bl.length()=186 2019-12-09T21:34:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.958+0000 7f709effd700 10 cache put: name=default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL info.flags=0x7 2019-12-09T21:34:24.954 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:24.958+0000 7f709effd700 10 moving default.rgw.meta+users.keys+ODWNEOWYVZRFIQZWQXXL to cache LRU end 2019-12-09T21:34:25.080 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2019-12-09T21:34:25.080 INFO:teuthology.orchestra.run.smithi067.stdout: "user_id": "bar.client.0", 2019-12-09T21:34:25.080 INFO:teuthology.orchestra.run.smithi067.stdout: "display_name": "Mr. bar.client.0", 2019-12-09T21:34:25.080 INFO:teuthology.orchestra.run.smithi067.stdout: "email": "bar.client.0+test@test.test", 2019-12-09T21:34:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "suspended": 0, 2019-12-09T21:34:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "max_buckets": 1000, 2019-12-09T21:34:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "subusers": [], 2019-12-09T21:34:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "keys": [ 2019-12-09T21:34:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-09T21:34:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "user": "bar.client.0", 2019-12-09T21:34:25.081 INFO:teuthology.orchestra.run.smithi067.stdout: "access_key": "ODWNEOWYVZRFIQZWQXXL", 2019-12-09T21:34:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "secret_key": "+MBXq46cTDvGI2vAaXv8QCY2g+3lAUcnYMvH6LwcX+Ue015CUWZ+FA==" 2019-12-09T21:34:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-09T21:34:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-09T21:34:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "swift_keys": [], 2019-12-09T21:34:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": [ 2019-12-09T21:34:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-09T21:34:25.082 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "user-policy", 2019-12-09T21:34:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "perm": "*" 2019-12-09T21:34:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-09T21:34:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-09T21:34:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "op_mask": "read, write, delete", 2019-12-09T21:34:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "", 2019-12-09T21:34:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "default_storage_class": "", 2019-12-09T21:34:25.083 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_tags": [], 2019-12-09T21:34:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_quota": { 2019-12-09T21:34:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T21:34:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T21:34:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T21:34:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T21:34:25.084 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T21:34:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T21:34:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "user_quota": { 2019-12-09T21:34:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T21:34:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T21:34:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T21:34:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T21:34:25.085 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T21:34:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T21:34:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "temp_url_keys": [], 2019-12-09T21:34:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "rgw", 2019-12-09T21:34:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: "mfa_ids": [] 2019-12-09T21:34:25.086 INFO:teuthology.orchestra.run.smithi067.stdout:} 2019-12-09T21:34:25.086 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:34:25.130 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.134+0000 7f70d0754ec0 20 remove_watcher() i=0 2019-12-09T21:34:25.131 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.134+0000 7f70d0754ec0 2 removed watcher, disabling cache 2019-12-09T21:34:25.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.178+0000 7f70d0754ec0 20 remove_watcher() i=1 2019-12-09T21:34:25.214 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.218+0000 7f70d0754ec0 20 remove_watcher() i=2 2019-12-09T21:34:25.282 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.285+0000 7f70d0754ec0 20 remove_watcher() i=3 2019-12-09T21:34:25.327 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.330+0000 7f70d0754ec0 20 remove_watcher() i=4 2019-12-09T21:34:25.379 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.383+0000 7f70d0754ec0 20 remove_watcher() i=5 2019-12-09T21:34:25.427 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.430+0000 7f70d0754ec0 20 remove_watcher() i=6 2019-12-09T21:34:25.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.467+0000 7f70d0754ec0 20 remove_watcher() i=7 2019-12-09T21:34:25.477 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid bar.client.0 --totp-serial 0934192586 --totp-seed ICZNUAPJC4ROS4TT7UVBQ65JIPOTYTPQRZVDO375L7HVPXIAL5LJKGALC7G3TOYV --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2019-12-09T21:34:25.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.669+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.691+0000 7f80a2089ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:25.687 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.691+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.700+0000 7f80a2089ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:25.696 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.700+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.704+0000 7f80a2089ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:25.701 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.704+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.708 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.712+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:25.887 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.892+0000 7f80a2089ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.892+0000 7f80a2089ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.892+0000 7f80a2089ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.892+0000 7f80a2089ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:34:25.888 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.892+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.894 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.899+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:25.894 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.899+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.905 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.910+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:25.905 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.910+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.913 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.917+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:25.913 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.917+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.921 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.925+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:25.921 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.926+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.930+0000 7f80a2089ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.930+0000 7f80a2089ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:34:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.930+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.930 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.935+0000 7f80a2089ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:25.931 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.935+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.943+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:25.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.943+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.951+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.951+0000 7f80a2089ec0 20 zonegroup default 2019-12-09T21:34:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.951+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.956+0000 7f80a2089ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:25.955 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.956+0000 7f80a2089ec0 10 Cannot find current period zone using local zone 2019-12-09T21:34:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.956+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.960+0000 7f80a2089ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:25.956 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.960+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.968+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:25.963 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.968+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:25.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.976+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:25.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.976+0000 7f80a2089ec0 20 zone default found 2019-12-09T21:34:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.976+0000 7f80a2089ec0 20 started sync module instance, tier type = 2019-12-09T21:34:25.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:25.976+0000 7f80a2089ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:34:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.344+0000 7f80a2089ec0 20 add_watcher() i=0 2019-12-09T21:34:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.344+0000 7f80a2089ec0 20 add_watcher() i=1 2019-12-09T21:34:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.344+0000 7f80a2089ec0 20 add_watcher() i=2 2019-12-09T21:34:26.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.344+0000 7f80a2089ec0 20 add_watcher() i=3 2019-12-09T21:34:26.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.344+0000 7f80a2089ec0 20 add_watcher() i=4 2019-12-09T21:34:26.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.344+0000 7f80a2089ec0 20 add_watcher() i=5 2019-12-09T21:34:26.341 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.344+0000 7f80a2089ec0 20 add_watcher() i=6 2019-12-09T21:34:26.375 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.380+0000 7f80a2089ec0 20 add_watcher() i=7 2019-12-09T21:34:26.376 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.380+0000 7f80a2089ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:34:26.376 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.380+0000 7f805ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:34:26.396 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.400+0000 7f808f2b3700 0 -- 172.21.15.67:0/3497802069 >> v2:172.21.15.81:3300/0 conn(0x563b4662a880 msgr2=0x563b4662cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:34:26.528 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:26.532+0000 7f808f2b3700 0 -- 172.21.15.67:0/3497802069 >> [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] conn(0x563b4663f4b0 msgr2=0x563b466418d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:34:27.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.003+0000 7f7fce7fc700 20 reqs_thread_entry: start 2019-12-09T21:34:27.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.003+0000 7f7fc7fff700 20 reqs_thread_entry: start 2019-12-09T21:34:27.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.003+0000 7f80a2089ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:34:27.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.003+0000 7f7fc67fc700 20 reqs_thread_entry: start 2019-12-09T21:34:28.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.004+0000 7f80a2089ec0 20 get_system_obj_state: rctx=0x7ffe415516d8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x563b466a5f50 s->prefetch_data=0 2019-12-09T21:34:28.000 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.004+0000 7f80a2089ec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2019-12-09T21:34:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.028+0000 7f80a2089ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2019-12-09T21:34:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.028+0000 7f80a2089ec0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-09T21:34:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.028+0000 7f80a2089ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T21:34:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.028+0000 7f80a2089ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T21:34:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.028+0000 7f80a2089ec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-09T21:34:28.024 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.028+0000 7f80a2089ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:28.047 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.051+0000 7f80a2089ec0 20 rados_obj.operate() r=0 bl.length=451 2019-12-09T21:34:28.047 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.051+0000 7f80a2089ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2019-12-09T21:34:28.047 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:28.051+0000 7f80a2089ec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-09T21:34:30.193 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:30.198+0000 7f80a2089ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-09T21:34:30.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:30.198+0000 7f80a2089ec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-09T21:34:30.194 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:30.198+0000 7f80a2089ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=650 2019-12-09T21:34:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:30.210+0000 7f8074ff9700 10 RGWWatcher::handle_notify() notify_id 601295421471 cookie 94812583800832 notifier 4980 bl.length()=650 2019-12-09T21:34:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:30.210+0000 7f8074ff9700 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-09T21:34:30.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:30.210+0000 7f8074ff9700 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-09T21:34:30.689 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:30.692+0000 7f80a2089ec0 20 remove_watcher() i=0 2019-12-09T21:34:30.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:30.692+0000 7f80a2089ec0 2 removed watcher, disabling cache 2019-12-09T21:34:31.145 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.149+0000 7f80a2089ec0 20 remove_watcher() i=1 2019-12-09T21:34:31.182 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.183+0000 7f80a2089ec0 20 remove_watcher() i=2 2019-12-09T21:34:31.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.240+0000 7f80a2089ec0 20 remove_watcher() i=3 2019-12-09T21:34:31.444 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.448+0000 7f80a2089ec0 20 remove_watcher() i=4 2019-12-09T21:34:31.548 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.553+0000 7f80a2089ec0 20 remove_watcher() i=5 2019-12-09T21:34:31.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.591+0000 7f80a2089ec0 20 remove_watcher() i=6 2019-12-09T21:34:31.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.620+0000 7f80a2089ec0 20 remove_watcher() i=7 2019-12-09T21:34:31.623 DEBUG:tasks.s3tests:Creating user foo.client.0 on client.0 2019-12-09T21:34:31.623 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key VNARFYYYGGEZXSJHFFIQ --secret U+4oGUNLARmatV/FW/TT8TFA0GL3stPnn+lV1lDtk5swG5tmHP3q+w== --email foo.client.0+test@test.test --caps 'user-policy=*' --cluster ceph 2019-12-09T21:34:31.742 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.746+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.770+0000 7fc0d3b57ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:31.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.770+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:31.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.779+0000 7fc0d3b57ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:31.774 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.779+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:31.779 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.783+0000 7fc0d3b57ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:31.779 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.783+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:31.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:31.791+0000 7fc0d3b57ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:32.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.008+0000 7fc0d3b57ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:34:32.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.008+0000 7fc0d3b57ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:34:32.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.008+0000 7fc0d3b57ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:34:32.005 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.008+0000 7fc0d3b57ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:34:32.006 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.008+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.024+0000 7fc0d3b57ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:32.025 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.024+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.030 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.033+0000 7fc0d3b57ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:32.030 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.033+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.041 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.045+0000 7fc0d3b57ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:32.041 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.045+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.056+0000 7fc0d3b57ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:32.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.056+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.058 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.062+0000 7fc0d3b57ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:32.058 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.062+0000 7fc0d3b57ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:34:32.058 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.062+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.068+0000 7fc0d3b57ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:32.064 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.068+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.075 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.079+0000 7fc0d3b57ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:32.075 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.079+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.088+0000 7fc0d3b57ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.088+0000 7fc0d3b57ec0 20 zonegroup default 2019-12-09T21:34:32.084 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.088+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.093+0000 7fc0d3b57ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.093+0000 7fc0d3b57ec0 10 Cannot find current period zone using local zone 2019-12-09T21:34:32.089 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.093+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.099+0000 7fc0d3b57ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:32.095 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.099+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.105 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.109+0000 7fc0d3b57ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:32.105 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.109+0000 7fc0d3b57ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:32.117 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.120+0000 7fc0d3b57ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:32.117 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.120+0000 7fc0d3b57ec0 20 zone default found 2019-12-09T21:34:32.118 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.120+0000 7fc0d3b57ec0 20 started sync module instance, tier type = 2019-12-09T21:34:32.118 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.120+0000 7fc0d3b57ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:34:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.620+0000 7fc0d3b57ec0 20 add_watcher() i=0 2019-12-09T21:34:32.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.620+0000 7fc0d3b57ec0 20 add_watcher() i=1 2019-12-09T21:34:32.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.620+0000 7fc0d3b57ec0 20 add_watcher() i=2 2019-12-09T21:34:32.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.620+0000 7fc0d3b57ec0 20 add_watcher() i=3 2019-12-09T21:34:32.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.620+0000 7fc0d3b57ec0 20 add_watcher() i=4 2019-12-09T21:34:32.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.620+0000 7fc0d3b57ec0 20 add_watcher() i=5 2019-12-09T21:34:32.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.628+0000 7fc0d3b57ec0 20 add_watcher() i=6 2019-12-09T21:34:32.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.670+0000 7fc0d3b57ec0 20 add_watcher() i=7 2019-12-09T21:34:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.670+0000 7fc0d3b57ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:34:32.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.670+0000 7fc013fff700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:34:32.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.678+0000 7fc0b8580700 0 -- 172.21.15.67:0/1884790999 >> v2:172.21.15.67:3300/0 conn(0x557cf3858090 msgr2=0x557cf385a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-09T21:34:32.735 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:32.739+0000 7fc0c1582700 0 -- 172.21.15.67:0/1151183561 >> [v2:172.21.15.67:6800/9311,v1:172.21.15.67:6801/9311] conn(0x7fc0000224b0 msgr2=0x7fc0000249a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-09T21:34:34.307 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.311+0000 7fbffe7fc700 20 reqs_thread_entry: start 2019-12-09T21:34:34.307 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.311+0000 7fbff7fff700 20 reqs_thread_entry: start 2019-12-09T21:34:34.307 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.311+0000 7fc0d3b57ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:34:34.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.311+0000 7fbff67fc700 20 reqs_thread_entry: start 2019-12-09T21:34:34.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.311+0000 7fc0d3b57ec0 20 get_system_obj_state: rctx=0x7ffd8b0a1c38 obj=default.rgw.meta:users.uid:foo.client.0 state=0x557cf38d5d90 s->prefetch_data=0 2019-12-09T21:34:34.308 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.312+0000 7fc0d3b57ec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2019-12-09T21:34:34.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.322+0000 7fc0d3b57ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2019-12-09T21:34:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.323+0000 7fc0d3b57ec0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-09T21:34:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.323+0000 7fc0d3b57ec0 20 get_system_obj_state: rctx=0x7ffd8b0a1c48 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x557cf38d7b20 s->prefetch_data=0 2019-12-09T21:34:34.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.323+0000 7fc0d3b57ec0 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2019-12-09T21:34:34.323 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.327+0000 7fc0d3b57ec0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2019-12-09T21:34:34.323 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.327+0000 7fc0d3b57ec0 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-09T21:34:34.323 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.327+0000 7fc0d3b57ec0 20 get_system_obj_state: rctx=0x7ffd8b0a1c48 obj=default.rgw.meta:users.keys:VNARFYYYGGEZXSJHFFIQ state=0x557cf38d7b20 s->prefetch_data=0 2019-12-09T21:34:34.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.327+0000 7fc0d3b57ec0 10 cache get: name=default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ : miss 2019-12-09T21:34:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.337+0000 7fc0d3b57ec0 10 cache put: name=default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ info.flags=0x0 2019-12-09T21:34:34.333 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.337+0000 7fc0d3b57ec0 10 adding default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ to cache LRU end 2019-12-09T21:34:34.334 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.337+0000 7fc0d3b57ec0 20 get_system_obj_state: rctx=0x7ffd8b0a1858 obj=default.rgw.meta:users.keys:VNARFYYYGGEZXSJHFFIQ state=0x557cf38d7b20 s->prefetch_data=0 2019-12-09T21:34:34.334 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.337+0000 7fc0d3b57ec0 10 cache get: name=default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ : type miss (requested=0x6, cached=0x0) 2019-12-09T21:34:34.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.346+0000 7fc0d3b57ec0 10 cache put: name=default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ info.flags=0x0 2019-12-09T21:34:34.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.346+0000 7fc0d3b57ec0 10 moving default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ to cache LRU end 2019-12-09T21:34:34.342 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.346+0000 7fc0d3b57ec0 20 get_system_obj_state: rctx=0x7ffd8b0a19d8 obj=default.rgw.meta:users.keys:VNARFYYYGGEZXSJHFFIQ state=0x557cf38d7b20 s->prefetch_data=0 2019-12-09T21:34:34.343 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.346+0000 7fc0d3b57ec0 10 cache get: name=default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ : type miss (requested=0x6, cached=0x0) 2019-12-09T21:34:34.348 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.352+0000 7fc0d3b57ec0 10 cache put: name=default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ info.flags=0x0 2019-12-09T21:34:34.348 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.352+0000 7fc0d3b57ec0 10 moving default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ to cache LRU end 2019-12-09T21:34:34.388 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.393+0000 7fc0d3b57ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-09T21:34:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.393+0000 7fc0d3b57ec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-09T21:34:34.389 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.393+0000 7fc0d3b57ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=636 2019-12-09T21:34:34.405 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.410+0000 7fc0b657c700 10 RGWWatcher::handle_notify() notify_id 605590388775 cookie 93995149790688 notifier 4601 bl.length()=636 2019-12-09T21:34:34.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.410+0000 7fc0b657c700 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-09T21:34:34.406 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:34.410+0000 7fc0b657c700 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-09T21:34:35.334 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.337+0000 7fc0d3b57ec0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2019-12-09T21:34:35.334 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.337+0000 7fc0d3b57ec0 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-09T21:34:35.334 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.337+0000 7fc0d3b57ec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=194 2019-12-09T21:34:35.352 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.355+0000 7fc0b657c700 10 RGWWatcher::handle_notify() notify_id 605590388779 cookie 93995149832384 notifier 4601 bl.length()=194 2019-12-09T21:34:35.352 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.355+0000 7fc0b657c700 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2019-12-09T21:34:35.352 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.355+0000 7fc0b657c700 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-09T21:34:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.880+0000 7fc0d3b57ec0 10 cache put: name=default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ info.flags=0x7 2019-12-09T21:34:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.881+0000 7fc0d3b57ec0 10 moving default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ to cache LRU end 2019-12-09T21:34:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.881+0000 7fc0d3b57ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=186 2019-12-09T21:34:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.892+0000 7fc0b657c700 10 RGWWatcher::handle_notify() notify_id 605590388808 cookie 93995149796000 notifier 4601 bl.length()=186 2019-12-09T21:34:35.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.893+0000 7fc0b657c700 10 cache put: name=default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ info.flags=0x7 2019-12-09T21:34:35.890 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:35.893+0000 7fc0b657c700 10 moving default.rgw.meta+users.keys+VNARFYYYGGEZXSJHFFIQ to cache LRU end 2019-12-09T21:34:36.303 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2019-12-09T21:34:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: "user_id": "foo.client.0", 2019-12-09T21:34:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: "display_name": "Mr. foo.client.0", 2019-12-09T21:34:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: "email": "foo.client.0+test@test.test", 2019-12-09T21:34:36.303 INFO:teuthology.orchestra.run.smithi067.stdout: "suspended": 0, 2019-12-09T21:34:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "max_buckets": 1000, 2019-12-09T21:34:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "subusers": [], 2019-12-09T21:34:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "keys": [ 2019-12-09T21:34:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-09T21:34:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "user": "foo.client.0", 2019-12-09T21:34:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "access_key": "VNARFYYYGGEZXSJHFFIQ", 2019-12-09T21:34:36.304 INFO:teuthology.orchestra.run.smithi067.stdout: "secret_key": "U+4oGUNLARmatV/FW/TT8TFA0GL3stPnn+lV1lDtk5swG5tmHP3q+w==" 2019-12-09T21:34:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-09T21:34:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-09T21:34:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "swift_keys": [], 2019-12-09T21:34:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": [ 2019-12-09T21:34:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-09T21:34:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "user-policy", 2019-12-09T21:34:36.305 INFO:teuthology.orchestra.run.smithi067.stdout: "perm": "*" 2019-12-09T21:34:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-09T21:34:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-09T21:34:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "op_mask": "read, write, delete", 2019-12-09T21:34:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "", 2019-12-09T21:34:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "default_storage_class": "", 2019-12-09T21:34:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_tags": [], 2019-12-09T21:34:36.306 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_quota": { 2019-12-09T21:34:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T21:34:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T21:34:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T21:34:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T21:34:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T21:34:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T21:34:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "user_quota": { 2019-12-09T21:34:36.307 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T21:34:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T21:34:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T21:34:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T21:34:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T21:34:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T21:34:36.308 INFO:teuthology.orchestra.run.smithi067.stdout: "temp_url_keys": [], 2019-12-09T21:34:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "rgw", 2019-12-09T21:34:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: "mfa_ids": [] 2019-12-09T21:34:36.309 INFO:teuthology.orchestra.run.smithi067.stdout:} 2019-12-09T21:34:36.309 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:34:36.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.377+0000 7fc0d3b57ec0 20 remove_watcher() i=0 2019-12-09T21:34:36.374 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.377+0000 7fc0d3b57ec0 2 removed watcher, disabling cache 2019-12-09T21:34:36.424 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.427+0000 7fc0d3b57ec0 20 remove_watcher() i=1 2019-12-09T21:34:36.478 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.481+0000 7fc0d3b57ec0 20 remove_watcher() i=2 2019-12-09T21:34:36.563 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.566+0000 7fc0d3b57ec0 20 remove_watcher() i=3 2019-12-09T21:34:36.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.620+0000 7fc0d3b57ec0 20 remove_watcher() i=4 2019-12-09T21:34:36.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.678+0000 7fc0d3b57ec0 20 remove_watcher() i=5 2019-12-09T21:34:36.732 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.735+0000 7fc0d3b57ec0 20 remove_watcher() i=6 2019-12-09T21:34:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.786+0000 7fc0d3b57ec0 20 remove_watcher() i=7 2019-12-09T21:34:36.792 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid foo.client.0 --totp-serial 5494954154 --totp-seed I23CBKF7A54G6HI676MNXD7CFWC6ZFWPIBAUCMNUF2TPH5NBIH3EBS3JV7B7TSVU --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2019-12-09T21:34:36.931 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.934+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.959+0000 7fa2f3af6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:36.955 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.959+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:36.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.969+0000 7fa2f3af6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:36.965 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.969+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.975+0000 7fa2f3af6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:36.972 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.975+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:36.984 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:36.987+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:37.135 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.138+0000 7fa2f3af6ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:34:37.135 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.138+0000 7fa2f3af6ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:34:37.135 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.138+0000 7fa2f3af6ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:34:37.135 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.138+0000 7fa2f3af6ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:34:37.135 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.138+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.143 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.146+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:37.143 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.146+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.155+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:37.152 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.155+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.162 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.166+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:37.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.166+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.178+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:37.175 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.179+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.182 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.186+0000 7fa2f3af6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:37.182 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.186+0000 7fa2f3af6ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:34:37.183 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.186+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.191+0000 7fa2f3af6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:37.188 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.191+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.196 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.200+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:37.196 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.200+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.212+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:37.208 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.212+0000 7fa2f3af6ec0 20 zonegroup default 2019-12-09T21:34:37.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.212+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.215 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.218+0000 7fa2f3af6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:37.215 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.218+0000 7fa2f3af6ec0 10 Cannot find current period zone using local zone 2019-12-09T21:34:37.215 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.218+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.220 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.224+0000 7fa2f3af6ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:37.220 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.224+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.231 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.235+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:37.232 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.235+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.243+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.243+0000 7fa2f3af6ec0 20 zone default found 2019-12-09T21:34:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.243+0000 7fa2f3af6ec0 20 started sync module instance, tier type = 2019-12-09T21:34:37.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.243+0000 7fa2f3af6ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:34:37.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.684+0000 7fa2f3af6ec0 20 add_watcher() i=0 2019-12-09T21:34:37.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.684+0000 7fa2f3af6ec0 20 add_watcher() i=1 2019-12-09T21:34:37.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.684+0000 7fa2f3af6ec0 20 add_watcher() i=2 2019-12-09T21:34:37.681 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.684+0000 7fa2f3af6ec0 20 add_watcher() i=3 2019-12-09T21:34:37.682 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.684+0000 7fa2f3af6ec0 20 add_watcher() i=4 2019-12-09T21:34:37.682 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.684+0000 7fa2f3af6ec0 20 add_watcher() i=5 2019-12-09T21:34:37.682 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.684+0000 7fa2f3af6ec0 20 add_watcher() i=6 2019-12-09T21:34:37.711 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.713+0000 7fa2f3af6ec0 20 add_watcher() i=7 2019-12-09T21:34:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.713+0000 7fa2f3af6ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:34:37.712 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.713+0000 7fa2b17e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:34:37.713 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.717+0000 7fa2e051f700 0 -- 172.21.15.67:0/3920982256 >> v2:172.21.15.67:3300/0 conn(0x556d772f9da0 msgr2=0x556d772fc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send injecting socket failure 2019-12-09T21:34:37.760 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:37.763+0000 7fa2e0d20700 0 -- 172.21.15.67:0/3814367114 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x556d7730ae70 msgr2=0x556d7730d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:34:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.327+0000 7fa2237fe700 20 reqs_thread_entry: start 2019-12-09T21:34:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.327+0000 7fa2217fa700 20 reqs_thread_entry: start 2019-12-09T21:34:39.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.327+0000 7fa2f3af6ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:34:39.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.328+0000 7fa20f7fe700 20 reqs_thread_entry: start 2019-12-09T21:34:39.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.328+0000 7fa2f3af6ec0 20 get_system_obj_state: rctx=0x7ffc45378ee8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x556d773778c0 s->prefetch_data=0 2019-12-09T21:34:39.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.328+0000 7fa2f3af6ec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2019-12-09T21:34:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.347+0000 7fa2f3af6ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2019-12-09T21:34:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.347+0000 7fa2f3af6ec0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-09T21:34:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.347+0000 7fa2f3af6ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T21:34:39.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.347+0000 7fa2f3af6ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T21:34:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.347+0000 7fa2f3af6ec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-09T21:34:39.345 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.347+0000 7fa2f3af6ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.369+0000 7fa2f3af6ec0 20 rados_obj.operate() r=0 bl.length=451 2019-12-09T21:34:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.369+0000 7fa2f3af6ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2019-12-09T21:34:39.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.369+0000 7fa2f3af6ec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-09T21:34:39.523 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.527+0000 7fa2f3af6ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-09T21:34:39.524 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.527+0000 7fa2f3af6ec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-09T21:34:39.524 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.527+0000 7fa2f3af6ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=650 2019-12-09T21:34:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.539+0000 7fa2d67fc700 10 RGWWatcher::handle_notify() notify_id 605590388776 cookie 93928639282464 notifier 5010 bl.length()=650 2019-12-09T21:34:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.539+0000 7fa2d67fc700 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-09T21:34:39.536 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.539+0000 7fa2d67fc700 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-09T21:34:39.953 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.957+0000 7fa2f3af6ec0 20 remove_watcher() i=0 2019-12-09T21:34:39.954 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:39.957+0000 7fa2f3af6ec0 2 removed watcher, disabling cache 2019-12-09T21:34:39.999 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.002+0000 7fa2f3af6ec0 20 remove_watcher() i=1 2019-12-09T21:34:40.035 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.039+0000 7fa2f3af6ec0 20 remove_watcher() i=2 2019-12-09T21:34:40.084 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.087+0000 7fa2f3af6ec0 20 remove_watcher() i=3 2019-12-09T21:34:40.145 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.149+0000 7fa2f3af6ec0 20 remove_watcher() i=4 2019-12-09T21:34:40.214 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.218+0000 7fa2f3af6ec0 20 remove_watcher() i=5 2019-12-09T21:34:40.272 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.276+0000 7fa2f3af6ec0 20 remove_watcher() i=6 2019-12-09T21:34:40.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.327+0000 7fa2f3af6ec0 20 remove_watcher() i=7 2019-12-09T21:34:40.331 DEBUG:tasks.s3tests:Creating user testx$tenanteduser.client.0 on client.0 2019-12-09T21:34:40.331 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid 'testx$tenanteduser.client.0' --display-name 'Mr. testx$tenanteduser.client.0' --access-key OCRIEMLAHEQGTBVTWNEP --secret 7Fi0r6hvUh2aMsDKOV3+QF450wc952tQ+AcB0WYy2Nh137pGNJAfiA== --email 'testx$tenanteduser.client.0+test@test.test' --caps 'user-policy=*' --cluster ceph 2019-12-09T21:34:40.483 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.487+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.514+0000 7f32cbe5fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:40.511 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.514+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.518 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.522+0000 7f32cbe5fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:40.518 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.522+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.523 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.526+0000 7f32cbe5fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:40.523 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.526+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.530 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.533+0000 7f32cbe5fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:40.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.742+0000 7f32cbe5fec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:34:40.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.742+0000 7f32cbe5fec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:34:40.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.742+0000 7f32cbe5fec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:34:40.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.742+0000 7f32cbe5fec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:34:40.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.742+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.752+0000 7f32cbe5fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:40.749 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.752+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.760+0000 7f32cbe5fec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:40.757 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.760+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.765 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.769+0000 7f32cbe5fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:40.766 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.769+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.777 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.780+0000 7f32cbe5fec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:40.777 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.780+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.786+0000 7f32cbe5fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.786+0000 7f32cbe5fec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:34:40.783 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.786+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.792+0000 7f32cbe5fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:40.789 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.792+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.796 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.799+0000 7f32cbe5fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:40.797 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.799+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.807+0000 7f32cbe5fec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.807+0000 7f32cbe5fec0 20 zonegroup default 2019-12-09T21:34:40.805 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.807+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.812+0000 7f32cbe5fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.812+0000 7f32cbe5fec0 10 Cannot find current period zone using local zone 2019-12-09T21:34:40.809 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.812+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.813 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.817+0000 7f32cbe5fec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:40.814 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.817+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.821 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.825+0000 7f32cbe5fec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:40.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.825+0000 7f32cbe5fec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:40.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.833+0000 7f32cbe5fec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:40.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.833+0000 7f32cbe5fec0 20 zone default found 2019-12-09T21:34:40.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.833+0000 7f32cbe5fec0 20 started sync module instance, tier type = 2019-12-09T21:34:40.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:40.833+0000 7f32cbe5fec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:34:41.234 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.238+0000 7f32cbe5fec0 20 add_watcher() i=0 2019-12-09T21:34:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.238+0000 7f32cbe5fec0 20 add_watcher() i=1 2019-12-09T21:34:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.238+0000 7f32cbe5fec0 20 add_watcher() i=2 2019-12-09T21:34:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.238+0000 7f32cbe5fec0 20 add_watcher() i=3 2019-12-09T21:34:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.238+0000 7f32cbe5fec0 20 add_watcher() i=4 2019-12-09T21:34:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.238+0000 7f32cbe5fec0 20 add_watcher() i=5 2019-12-09T21:34:41.235 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.238+0000 7f32cbe5fec0 20 add_watcher() i=6 2019-12-09T21:34:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.273+0000 7f32b9089700 0 -- 172.21.15.67:0/2572449699 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x555f7edb0370 msgr2=0x555f7edb2790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:34:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.273+0000 7f32cbe5fec0 20 add_watcher() i=7 2019-12-09T21:34:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.273+0000 7f32cbe5fec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:34:41.270 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.273+0000 7f3289fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:34:41.290 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:41.293+0000 7f32b988a700 0 -- 172.21.15.67:0/1927478939 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x555f7edc4dd0 msgr2=0x555f7edc51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:34:42.913 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.916+0000 7f31fbfff700 20 reqs_thread_entry: start 2019-12-09T21:34:42.913 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.916+0000 7f31f9ffb700 20 reqs_thread_entry: start 2019-12-09T21:34:42.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.916+0000 7f32cbe5fec0 20 init_complete bucket index max shards: 0 2019-12-09T21:34:42.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.916+0000 7f31e7fff700 20 reqs_thread_entry: start 2019-12-09T21:34:42.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.916+0000 7f32cbe5fec0 20 get_system_obj_state: rctx=0x7ffc12d29908 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x555f7ee42680 s->prefetch_data=0 2019-12-09T21:34:42.914 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.916+0000 7f32cbe5fec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2019-12-09T21:34:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.922+0000 7f32cbe5fec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x0 2019-12-09T21:34:42.919 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.922+0000 7f32cbe5fec0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-09T21:34:42.920 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.922+0000 7f32cbe5fec0 20 get_system_obj_state: rctx=0x7ffc12d29918 obj=default.rgw.meta:users.email:testx$tenanteduser.client.0+test@test.test state=0x555f7ee44430 s->prefetch_data=0 2019-12-09T21:34:42.920 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.922+0000 7f32cbe5fec0 10 cache get: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test : miss 2019-12-09T21:34:42.924 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.927+0000 7f32cbe5fec0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x0 2019-12-09T21:34:42.924 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.927+0000 7f32cbe5fec0 10 adding default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2019-12-09T21:34:42.924 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.927+0000 7f32cbe5fec0 20 get_system_obj_state: rctx=0x7ffc12d29918 obj=default.rgw.meta:users.keys:OCRIEMLAHEQGTBVTWNEP state=0x555f7ee44430 s->prefetch_data=0 2019-12-09T21:34:42.925 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.927+0000 7f32cbe5fec0 10 cache get: name=default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP : miss 2019-12-09T21:34:42.934 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.937+0000 7f32cbe5fec0 10 cache put: name=default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP info.flags=0x0 2019-12-09T21:34:42.934 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.937+0000 7f32cbe5fec0 10 adding default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP to cache LRU end 2019-12-09T21:34:42.934 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.937+0000 7f32cbe5fec0 20 get_system_obj_state: rctx=0x7ffc12d29528 obj=default.rgw.meta:users.keys:OCRIEMLAHEQGTBVTWNEP state=0x555f7ee44430 s->prefetch_data=0 2019-12-09T21:34:42.934 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.937+0000 7f32cbe5fec0 10 cache get: name=default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP : type miss (requested=0x6, cached=0x0) 2019-12-09T21:34:42.943 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.946+0000 7f32cbe5fec0 10 cache put: name=default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP info.flags=0x0 2019-12-09T21:34:42.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.946+0000 7f32cbe5fec0 10 moving default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP to cache LRU end 2019-12-09T21:34:42.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.946+0000 7f32cbe5fec0 20 get_system_obj_state: rctx=0x7ffc12d296a8 obj=default.rgw.meta:users.keys:OCRIEMLAHEQGTBVTWNEP state=0x555f7ee44430 s->prefetch_data=0 2019-12-09T21:34:42.944 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.946+0000 7f32cbe5fec0 10 cache get: name=default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP : type miss (requested=0x6, cached=0x0) 2019-12-09T21:34:42.952 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.955+0000 7f32cbe5fec0 10 cache put: name=default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP info.flags=0x0 2019-12-09T21:34:42.952 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:42.955+0000 7f32cbe5fec0 10 moving default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP to cache LRU end 2019-12-09T21:34:43.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.031+0000 7f32cbe5fec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2019-12-09T21:34:43.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.031+0000 7f32cbe5fec0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-09T21:34:43.028 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.031+0000 7f32cbe5fec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=710 2019-12-09T21:34:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.042+0000 7f329a7fc700 10 RGWWatcher::handle_notify() notify_id 605590388809 cookie 93868638530720 notifier 4613 bl.length()=710 2019-12-09T21:34:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.042+0000 7f329a7fc700 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2019-12-09T21:34:43.040 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.042+0000 7f329a7fc700 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-09T21:34:43.102 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.106+0000 7f32cbe5fec0 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2019-12-09T21:34:43.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.106+0000 7f32cbe5fec0 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2019-12-09T21:34:43.103 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.106+0000 7f32cbe5fec0 10 distributing notification oid=default.rgw.control:notify.7 bl.length()=224 2019-12-09T21:34:43.124 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.127+0000 7f329a7fc700 10 RGWWatcher::handle_notify() notify_id 605590388780 cookie 93868638547152 notifier 4613 bl.length()=224 2019-12-09T21:34:43.124 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.127+0000 7f329a7fc700 10 cache put: name=default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test info.flags=0x7 2019-12-09T21:34:43.124 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.127+0000 7f329a7fc700 10 moving default.rgw.meta+users.email+testx$tenanteduser.client.0+test@test.test to cache LRU end 2019-12-09T21:34:43.243 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.245+0000 7f32cbe5fec0 10 cache put: name=default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP info.flags=0x7 2019-12-09T21:34:43.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.245+0000 7f32cbe5fec0 10 moving default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP to cache LRU end 2019-12-09T21:34:43.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.246+0000 7f32cbe5fec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=201 2019-12-09T21:34:43.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.259+0000 7f329a7fc700 10 RGWWatcher::handle_notify() notify_id 605590388777 cookie 93868638504832 notifier 4613 bl.length()=201 2019-12-09T21:34:43.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.259+0000 7f329a7fc700 10 cache put: name=default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP info.flags=0x7 2019-12-09T21:34:43.256 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.259+0000 7f329a7fc700 10 moving default.rgw.meta+users.keys+OCRIEMLAHEQGTBVTWNEP to cache LRU end 2019-12-09T21:34:43.333 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2019-12-09T21:34:43.333 INFO:teuthology.orchestra.run.smithi067.stdout: "user_id": "testx$tenanteduser.client.0", 2019-12-09T21:34:43.333 INFO:teuthology.orchestra.run.smithi067.stdout: "display_name": "Mr. testx$tenanteduser.client.0", 2019-12-09T21:34:43.333 INFO:teuthology.orchestra.run.smithi067.stdout: "email": "testx$tenanteduser.client.0+test@test.test", 2019-12-09T21:34:43.334 INFO:teuthology.orchestra.run.smithi067.stdout: "suspended": 0, 2019-12-09T21:34:43.334 INFO:teuthology.orchestra.run.smithi067.stdout: "max_buckets": 1000, 2019-12-09T21:34:43.334 INFO:teuthology.orchestra.run.smithi067.stdout: "subusers": [], 2019-12-09T21:34:43.334 INFO:teuthology.orchestra.run.smithi067.stdout: "keys": [ 2019-12-09T21:34:43.334 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-09T21:34:43.335 INFO:teuthology.orchestra.run.smithi067.stdout: "user": "testx$tenanteduser.client.0", 2019-12-09T21:34:43.335 INFO:teuthology.orchestra.run.smithi067.stdout: "access_key": "OCRIEMLAHEQGTBVTWNEP", 2019-12-09T21:34:43.335 INFO:teuthology.orchestra.run.smithi067.stdout: "secret_key": "7Fi0r6hvUh2aMsDKOV3+QF450wc952tQ+AcB0WYy2Nh137pGNJAfiA==" 2019-12-09T21:34:43.335 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-09T21:34:43.335 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-09T21:34:43.335 INFO:teuthology.orchestra.run.smithi067.stdout: "swift_keys": [], 2019-12-09T21:34:43.336 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": [ 2019-12-09T21:34:43.336 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-09T21:34:43.336 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "user-policy", 2019-12-09T21:34:43.336 INFO:teuthology.orchestra.run.smithi067.stdout: "perm": "*" 2019-12-09T21:34:43.336 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-09T21:34:43.337 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-09T21:34:43.337 INFO:teuthology.orchestra.run.smithi067.stdout: "op_mask": "read, write, delete", 2019-12-09T21:34:43.337 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "", 2019-12-09T21:34:43.337 INFO:teuthology.orchestra.run.smithi067.stdout: "default_storage_class": "", 2019-12-09T21:34:43.338 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_tags": [], 2019-12-09T21:34:43.338 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_quota": { 2019-12-09T21:34:43.338 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T21:34:43.338 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T21:34:43.338 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T21:34:43.338 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T21:34:43.338 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T21:34:43.339 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T21:34:43.339 INFO:teuthology.orchestra.run.smithi067.stdout: "user_quota": { 2019-12-09T21:34:43.339 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-09T21:34:43.339 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-09T21:34:43.339 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-09T21:34:43.339 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-09T21:34:43.339 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-09T21:34:43.340 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-09T21:34:43.340 INFO:teuthology.orchestra.run.smithi067.stdout: "temp_url_keys": [], 2019-12-09T21:34:43.340 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "rgw", 2019-12-09T21:34:43.340 INFO:teuthology.orchestra.run.smithi067.stdout: "mfa_ids": [] 2019-12-09T21:34:43.340 INFO:teuthology.orchestra.run.smithi067.stdout:} 2019-12-09T21:34:43.341 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:34:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.396+0000 7f32cbe5fec0 20 remove_watcher() i=0 2019-12-09T21:34:43.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.396+0000 7f32cbe5fec0 2 removed watcher, disabling cache 2019-12-09T21:34:43.447 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.446+0000 7f32cbe5fec0 20 remove_watcher() i=1 2019-12-09T21:34:43.479 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.482+0000 7f32cbe5fec0 20 remove_watcher() i=2 2019-12-09T21:34:43.543 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.546+0000 7f32cbe5fec0 20 remove_watcher() i=3 2019-12-09T21:34:43.581 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.584+0000 7f32cbe5fec0 20 remove_watcher() i=4 2019-12-09T21:34:43.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.680+0000 7f32cbe5fec0 20 remove_watcher() i=5 2019-12-09T21:34:43.727 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.730+0000 7f32cbe5fec0 20 remove_watcher() i=6 2019-12-09T21:34:43.781 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.784+0000 7f32cbe5fec0 20 remove_watcher() i=7 2019-12-09T21:34:43.788 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 mfa create --uid 'testx$tenanteduser.client.0' --totp-serial 9973825810 --totp-seed NFDT3GFXOVTCUJFX674A5X2UTCCA7GPV272LGMWQZDAV7EAE3YEL33T6ACSCSTRV --totp-seconds 5 --totp-window 8 --totp-seed-type base32 --cluster ceph 2019-12-09T21:34:43.935 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.938+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:43.970 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.974+0000 7f3147d92ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:43.971 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.974+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:43.987 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.985+0000 7f3147d92ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:43.987 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.985+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:43.988 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.990+0000 7f3147d92ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:43.988 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:43.990+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:43.996 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.000+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.204+0000 7f3147d92ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-09T21:34:44.201 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.204+0000 7f3147d92ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-09T21:34:44.202 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.204+0000 7f3147d92ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-09T21:34:44.202 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.204+0000 7f3147d92ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-09T21:34:44.202 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.204+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.212+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:44.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.212+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.224+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:44.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.224+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.232 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.235+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:44.232 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.235+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.242 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.245+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:44.242 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.245+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.250+0000 7f3147d92ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:44.247 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.250+0000 7f3147d92ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-09T21:34:44.248 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.250+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.252 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.256+0000 7f3147d92ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:44.253 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.256+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.264+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:44.261 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.264+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.276+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-09T21:34:44.273 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.276+0000 7f3147d92ec0 20 zonegroup default 2019-12-09T21:34:44.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.276+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.304 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.308+0000 7f3147d92ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:44.305 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.308+0000 7f3147d92ec0 10 Cannot find current period zone using local zone 2019-12-09T21:34:44.305 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.308+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.321 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.325+0000 7f3147d92ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-09T21:34:44.322 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.325+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.334 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.338+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-09T21:34:44.335 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.338+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:44.347 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.350+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-09T21:34:44.347 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.350+0000 7f3147d92ec0 20 zone default found 2019-12-09T21:34:44.347 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.350+0000 7f3147d92ec0 20 started sync module instance, tier type = 2019-12-09T21:34:44.347 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.350+0000 7f3147d92ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-09T21:34:44.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.700+0000 7f3147d92ec0 20 add_watcher() i=0 2019-12-09T21:34:44.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.700+0000 7f3147d92ec0 20 add_watcher() i=1 2019-12-09T21:34:44.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.700+0000 7f3147d92ec0 20 add_watcher() i=2 2019-12-09T21:34:44.697 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.700+0000 7f3147d92ec0 20 add_watcher() i=3 2019-12-09T21:34:44.698 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.700+0000 7f3147d92ec0 20 add_watcher() i=4 2019-12-09T21:34:44.698 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.700+0000 7f3147d92ec0 20 add_watcher() i=5 2019-12-09T21:34:44.702 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.705+0000 7f3147d92ec0 20 add_watcher() i=6 2019-12-09T21:34:44.735 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.739+0000 7f3147d92ec0 20 add_watcher() i=7 2019-12-09T21:34:44.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.739+0000 7f3147d92ec0 2 all 8 watchers are set, enabling cache 2019-12-09T21:34:44.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.739+0000 7f31057e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-09T21:34:44.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.741+0000 7f3134fbc700 0 -- 172.21.15.67:0/1619496488 >> v2:172.21.15.67:3300/0 conn(0x5568f2f15320 msgr2=0x5568f2f17740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0)._try_send injecting socket failure 2019-12-09T21:34:44.786 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:44.789+0000 7f3134fbc700 0 -- 172.21.15.67:0/638460252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x5568f2f1c7b0 msgr2=0x5568f2f1ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-09T21:34:46.190 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.193+0000 7f30777fe700 20 reqs_thread_entry: start 2019-12-09T21:34:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.194+0000 7f30757fa700 20 reqs_thread_entry: start 2019-12-09T21:34:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.194+0000 7f3147d92ec0 20 init_complete bucket index max shards: 0 2019-12-09T21:34:46.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.194+0000 7f30637fe700 20 reqs_thread_entry: start 2019-12-09T21:34:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.194+0000 7f3147d92ec0 20 get_system_obj_state: rctx=0x7ffc016e9b38 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x5568f2f8d880 s->prefetch_data=0 2019-12-09T21:34:46.192 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.194+0000 7f3147d92ec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2019-12-09T21:34:46.202 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.205+0000 7f3147d92ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2019-12-09T21:34:46.202 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.205+0000 7f3147d92ec0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-09T21:34:46.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.205+0000 7f3147d92ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-09T21:34:46.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.205+0000 7f3147d92ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-09T21:34:46.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.205+0000 7f3147d92ec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-09T21:34:46.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.205+0000 7f3147d92ec0 20 rados->read ofs=0 len=0 2019-12-09T21:34:46.216 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.219+0000 7f3147d92ec0 20 rados_obj.operate() r=0 bl.length=510 2019-12-09T21:34:46.216 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.219+0000 7f3147d92ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2019-12-09T21:34:46.217 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.219+0000 7f3147d92ec0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-09T21:34:46.370 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.373+0000 7f3147d92ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2019-12-09T21:34:46.370 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.373+0000 7f3147d92ec0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-09T21:34:46.371 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.373+0000 7f3147d92ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=724 2019-12-09T21:34:46.380 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.383+0000 7f312a7fc700 10 RGWWatcher::handle_notify() notify_id 605590388810 cookie 93909240780480 notifier 4628 bl.length()=724 2019-12-09T21:34:46.380 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.384+0000 7f312a7fc700 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x17 2019-12-09T21:34:46.381 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.384+0000 7f312a7fc700 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-09T21:34:46.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.669+0000 7f3147d92ec0 20 remove_watcher() i=0 2019-12-09T21:34:46.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.669+0000 7f3147d92ec0 2 removed watcher, disabling cache 2019-12-09T21:34:46.718 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.721+0000 7f3147d92ec0 20 remove_watcher() i=1 2019-12-09T21:34:46.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.759+0000 7f3147d92ec0 20 remove_watcher() i=2 2019-12-09T21:34:46.804 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.807+0000 7f3147d92ec0 20 remove_watcher() i=3 2019-12-09T21:34:46.841 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.844+0000 7f3147d92ec0 20 remove_watcher() i=4 2019-12-09T21:34:46.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.892+0000 7f3147d92ec0 20 remove_watcher() i=5 2019-12-09T21:34:46.933 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.936+0000 7f3147d92ec0 20 remove_watcher() i=6 2019-12-09T21:34:46.979 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-09T21:34:46.982+0000 7f3147d92ec0 20 remove_watcher() i=7 2019-12-09T21:34:46.990 INFO:tasks.s3tests:Configuring s3-tests... 2019-12-09T21:34:46.991 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest/s3-tests && ./bootstrap 2019-12-09T21:34:53.067 INFO:teuthology.orchestra.run.smithi067.stderr:./bootstrap: missing required RPM packages. Installing via sudo. 2019-12-09T21:34:53.175 INFO:teuthology.orchestra.run.smithi067.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-09T21:34:53.217 INFO:teuthology.orchestra.run.smithi067.stdout:Loading mirror speeds from cached hostfile 2019-12-09T21:34:53.217 INFO:teuthology.orchestra.run.smithi067.stdout: * base: mirror.ash.fastserv.com 2019-12-09T21:34:53.218 INFO:teuthology.orchestra.run.smithi067.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-09T21:34:53.218 INFO:teuthology.orchestra.run.smithi067.stdout: * extras: mirror.math.princeton.edu 2019-12-09T21:34:53.219 INFO:teuthology.orchestra.run.smithi067.stdout: * updates: mirror.jaleco.com 2019-12-09T21:34:53.895 INFO:teuthology.orchestra.run.smithi067.stdout:7 packages excluded due to repository priority protections 2019-12-09T21:34:54.295 INFO:teuthology.orchestra.run.smithi067.stdout:Resolving Dependencies 2019-12-09T21:34:54.296 INFO:teuthology.orchestra.run.smithi067.stdout:--> Running transaction check 2019-12-09T21:34:54.296 INFO:teuthology.orchestra.run.smithi067.stdout:---> Package libffi-devel.x86_64 0:3.0.13-18.el7 will be installed 2019-12-09T21:34:54.705 INFO:teuthology.orchestra.run.smithi067.stdout:--> Finished Dependency Resolution 2019-12-09T21:34:54.943 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:34:54.943 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies Resolved 2019-12-09T21:34:54.944 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:34:54.945 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T21:34:54.945 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2019-12-09T21:34:54.945 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T21:34:54.945 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2019-12-09T21:34:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: libffi-devel x86_64 3.0.13-18.el7 base 23 k 2019-12-09T21:34:54.946 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:34:54.946 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2019-12-09T21:34:54.946 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2019-12-09T21:34:54.946 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2019-12-09T21:34:54.947 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:34:54.947 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 23 k 2019-12-09T21:34:54.947 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 27 k 2019-12-09T21:34:54.947 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading packages: 2019-12-09T21:34:55.027 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2019-12-09T21:34:55.032 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2019-12-09T21:34:55.034 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded 2019-12-09T21:34:55.035 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2019-12-09T21:34:55.516 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libffi-devel-3.0.13-18.el7.x86_64 1/1 2019-12-09T21:34:56.042 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libffi-devel-3.0.13-18.el7.x86_64 1/1 2019-12-09T21:34:56.042 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:34:56.042 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2019-12-09T21:34:56.042 INFO:teuthology.orchestra.run.smithi067.stdout: libffi-devel.x86_64 0:3.0.13-18.el7 2019-12-09T21:34:56.043 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:34:56.043 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2019-12-09T21:34:56.251 INFO:teuthology.orchestra.run.smithi067.stdout:New python executable in /home/ubuntu/cephtest/s3-tests/virtualenv/bin/python 2019-12-09T21:34:57.605 INFO:teuthology.orchestra.run.smithi067.stdout:Installing setuptools, pip, wheel...done. 2019-12-09T21:34:57.939 INFO:teuthology.orchestra.run.smithi067.stderr:Cache entry deserialization failed, entry ignored 2019-12-09T21:34:58.107 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting pip 2019-12-09T21:34:58.184 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl 2019-12-09T21:34:58.254 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: pip 2019-12-09T21:34:58.254 INFO:teuthology.orchestra.run.smithi067.stdout: Found existing installation: pip 9.0.1 2019-12-09T21:34:58.339 INFO:teuthology.orchestra.run.smithi067.stdout: Uninstalling pip-9.0.1: 2019-12-09T21:34:58.419 INFO:teuthology.orchestra.run.smithi067.stdout: Successfully uninstalled pip-9.0.1 2019-12-09T21:34:59.071 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed pip-19.3.1 2019-12-09T21:34:59.359 INFO:teuthology.orchestra.run.smithi067.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-09T21:34:59.859 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting setuptools==32.3.1 2019-12-09T21:34:59.862 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/69/19/b1dff551058ce79d88b1e3688f1c735590d7ddf44d10681512133b35019f/setuptools-32.3.1-py2.py3-none-any.whl 2019-12-09T21:34:59.905 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: setuptools 2019-12-09T21:34:59.906 INFO:teuthology.orchestra.run.smithi067.stdout: Found existing installation: setuptools 28.8.0 2019-12-09T21:34:59.921 INFO:teuthology.orchestra.run.smithi067.stdout: Uninstalling setuptools-28.8.0: 2019-12-09T21:34:59.930 INFO:teuthology.orchestra.run.smithi067.stdout: Successfully uninstalled setuptools-28.8.0 2019-12-09T21:35:00.058 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed setuptools-32.3.1 2019-12-09T21:35:00.570 INFO:teuthology.orchestra.run.smithi067.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-09T21:35:00.774 INFO:teuthology.orchestra.run.smithi067.stdout:Processing /home/ubuntu/.cache/pip/wheels/54/b7/c7/2ada654ee54483c9329871665aaf4a6056c3ce36f29cf66e67/PyYAML-5.2-cp27-cp27mu-linux_x86_64.whl 2019-12-09T21:35:00.835 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting nose>=1.0.0 2019-12-09T21:35:00.838 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl 2019-12-09T21:35:01.021 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting boto>=2.6.0 2019-12-09T21:35:01.036 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl 2019-12-09T21:35:01.634 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting boto3>=1.0.0 2019-12-09T21:35:01.691 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/b7/f7/45cb1f72e7824d2b16b5232b914c8a4f490ae83db569f28023a1ebdea52c/boto3-1.10.35-py2.py3-none-any.whl (128kB) 2019-12-09T21:35:01.846 INFO:teuthology.orchestra.run.smithi067.stdout:Processing /home/ubuntu/.cache/pip/wheels/56/0f/19/fbbf81e5764e6d8b74501c4357a88c14c94466ec777c03734c/bunch-1.0.1-cp27-none-any.whl 2019-12-09T21:35:02.138 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting gevent>=1.0 2019-12-09T21:35:02.151 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/e6/e9/3a693414f706e12abe60554cd73c5ae8f848b182ae58018f93d86c9eb418/gevent-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-09T21:35:02.437 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting isodate>=0.4.4 2019-12-09T21:35:02.450 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl (45kB) 2019-12-09T21:35:02.573 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting requests>=0.14.0 2019-12-09T21:35:02.593 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB) 2019-12-09T21:35:02.896 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting pytz>=2011k 2019-12-09T21:35:02.906 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl (509kB) 2019-12-09T21:35:03.131 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting ordereddict 2019-12-09T21:35:03.142 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/53/25/ef88e8e45db141faa9598fbf7ad0062df8f50f881a36ed6a0073e1572126/ordereddict-1.1.tar.gz 2019-12-09T21:35:03.465 INFO:teuthology.orchestra.run.smithi067.stdout:Processing /home/ubuntu/.cache/pip/wheels/51/ab/65/f318b4271019823cf7d7c178ac76a8ea4d6e6271fe9f5ba488/httplib2-0.14.0-cp27-none-any.whl 2019-12-09T21:35:03.832 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting lxml 2019-12-09T21:35:03.853 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/25/73/730ad7249847f741c7e622f52b971daa5540f6fb87589bc92c717f5aafba/lxml-4.4.2-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-09T21:35:04.050 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting jmespath<1.0.0,>=0.7.1 2019-12-09T21:35:04.060 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl 2019-12-09T21:35:04.705 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting botocore<1.14.0,>=1.13.35 2019-12-09T21:35:04.714 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/1a/69/abe9276bc3699aa89cd5f743982aa9d9ce929d7f95b4900bba60922747b0/botocore-1.13.35-py2.py3-none-any.whl (5.8MB) 2019-12-09T21:35:05.332 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting s3transfer<0.3.0,>=0.2.0 2019-12-09T21:35:05.342 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl (70kB) 2019-12-09T21:35:05.543 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting greenlet>=0.4.14; platform_python_implementation == "CPython" 2019-12-09T21:35:05.545 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/8b/6e/f2d25875713ad0885c8d3c69269697406652e6f64e1a6bd8264f7a609327/greenlet-0.4.15-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-09T21:35:05.623 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting six 2019-12-09T21:35:05.633 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl 2019-12-09T21:35:05.733 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting chardet<3.1.0,>=3.0.2 2019-12-09T21:35:05.745 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB) 2019-12-09T21:35:05.853 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting idna<2.9,>=2.5 2019-12-09T21:35:05.864 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB) 2019-12-09T21:35:06.001 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 2019-12-09T21:35:06.010 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/b4/40/a9837291310ee1ccc242ceb6ebfd9eb21539649f193a7c8c86ba15b98539/urllib3-1.25.7-py2.py3-none-any.whl (125kB) 2019-12-09T21:35:06.157 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting certifi>=2017.4.17 2019-12-09T21:35:06.172 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/b9/63/df50cac98ea0d5b006c55a399c3bf1db9da7b5a24de7890bc9cfd5dd9e99/certifi-2019.11.28-py2.py3-none-any.whl (156kB) 2019-12-09T21:35:06.259 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting docutils<0.16,>=0.10 2019-12-09T21:35:06.270 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/3a/dc/bf2b15d1fa15a6f7a9e77a61b74ecbbae7258558fcda8ffc9a6638a6b327/docutils-0.15.2-py2-none-any.whl (548kB) 2019-12-09T21:35:06.410 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting python-dateutil<2.8.1,>=2.1; python_version >= "2.7" 2019-12-09T21:35:06.420 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB) 2019-12-09T21:35:06.547 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" 2019-12-09T21:35:06.557 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/d8/a6/f46ae3f1da0cd4361c344888f59ec2f5785e69c872e175a748ef6071cdb5/futures-3.3.0-py2-none-any.whl 2019-12-09T21:35:06.589 INFO:teuthology.orchestra.run.smithi067.stdout:Building wheels for collected packages: ordereddict 2019-12-09T21:35:06.590 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for ordereddict (setup.py): started 2019-12-09T21:35:06.838 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for ordereddict (setup.py): finished with status 'done' 2019-12-09T21:35:06.838 INFO:teuthology.orchestra.run.smithi067.stdout: Created wheel for ordereddict: filename=ordereddict-1.1-cp27-none-any.whl size=3327 sha256=4c408f9124d241cdada65a41c76d0001f90a17f355f516d1fec9b32816f98ae1 2019-12-09T21:35:06.838 INFO:teuthology.orchestra.run.smithi067.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/78/d3/a8/9f52c7a389a0ffba6f575a3886b5f7a55461fb6ec34aa5fd38 2019-12-09T21:35:06.841 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully built ordereddict 2019-12-09T21:35:06.926 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: PyYAML, nose, boto, jmespath, docutils, six, python-dateutil, urllib3, botocore, futures, s3transfer, boto3, bunch, greenlet, gevent, isodate, chardet, idna, certifi, requests, pytz, ordereddict, httplib2, lxml 2019-12-09T21:35:09.274 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed PyYAML-5.2 boto-2.49.0 boto3-1.10.35 botocore-1.13.35 bunch-1.0.1 certifi-2019.11.28 chardet-3.0.4 docutils-0.15.2 futures-3.3.0 gevent-1.4.0 greenlet-0.4.15 httplib2-0.14.0 idna-2.8 isodate-0.6.0 jmespath-0.9.4 lxml-4.4.2 nose-1.3.7 ordereddict-1.1 python-dateutil-2.8.0 pytz-2019.3 requests-2.22.0 s3transfer-0.2.1 six-1.13.0 urllib3-1.25.7 2019-12-09T21:35:09.595 INFO:teuthology.orchestra.run.smithi067.stdout:running develop 2019-12-09T21:35:09.652 INFO:teuthology.orchestra.run.smithi067.stdout:running egg_info 2019-12-09T21:35:09.653 INFO:teuthology.orchestra.run.smithi067.stdout:creating s3tests.egg-info 2019-12-09T21:35:09.656 INFO:teuthology.orchestra.run.smithi067.stdout:writing requirements to s3tests.egg-info/requires.txt 2019-12-09T21:35:09.656 INFO:teuthology.orchestra.run.smithi067.stdout:writing s3tests.egg-info/PKG-INFO 2019-12-09T21:35:09.657 INFO:teuthology.orchestra.run.smithi067.stdout:writing top-level names to s3tests.egg-info/top_level.txt 2019-12-09T21:35:09.657 INFO:teuthology.orchestra.run.smithi067.stdout:writing dependency_links to s3tests.egg-info/dependency_links.txt 2019-12-09T21:35:09.658 INFO:teuthology.orchestra.run.smithi067.stdout:writing entry points to s3tests.egg-info/entry_points.txt 2019-12-09T21:35:09.658 INFO:teuthology.orchestra.run.smithi067.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2019-12-09T21:35:09.658 INFO:teuthology.orchestra.run.smithi067.stdout:reading manifest file 's3tests.egg-info/SOURCES.txt' 2019-12-09T21:35:09.659 INFO:teuthology.orchestra.run.smithi067.stdout:writing manifest file 's3tests.egg-info/SOURCES.txt' 2019-12-09T21:35:09.660 INFO:teuthology.orchestra.run.smithi067.stdout:running build_ext 2019-12-09T21:35:09.660 INFO:teuthology.orchestra.run.smithi067.stdout:Creating /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/s3tests.egg-link (link to .) 2019-12-09T21:35:09.660 INFO:teuthology.orchestra.run.smithi067.stdout:Adding s3tests 0.0.1 to easy-install.pth file 2019-12-09T21:35:09.663 INFO:teuthology.orchestra.run.smithi067.stdout:Installing s3tests-fuzz-headers script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-09T21:35:09.663 INFO:teuthology.orchestra.run.smithi067.stdout:Installing s3tests-generate-objects script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-09T21:35:09.663 INFO:teuthology.orchestra.run.smithi067.stdout:Installing s3tests-test-roundtrip script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-09T21:35:09.664 INFO:teuthology.orchestra.run.smithi067.stdout:Installing s3tests-analysis-rwstats script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-09T21:35:09.664 INFO:teuthology.orchestra.run.smithi067.stdout:Installing s3tests-test-readwrite script to /home/ubuntu/cephtest/s3-tests/virtualenv/bin 2019-12-09T21:35:09.664 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.664 INFO:teuthology.orchestra.run.smithi067.stdout:Installed /home/ubuntu/cephtest/s3-tests 2019-12-09T21:35:09.665 INFO:teuthology.orchestra.run.smithi067.stdout:Processing dependencies for s3tests==0.0.1 2019-12-09T21:35:09.718 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for isodate==0.6.0 2019-12-09T21:35:09.718 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: isodate 0.6.0 2019-12-09T21:35:09.718 INFO:teuthology.orchestra.run.smithi067.stdout:Adding isodate 0.6.0 to easy-install.pth file 2019-12-09T21:35:09.718 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.718 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.719 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for gevent==1.4.0 2019-12-09T21:35:09.719 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: gevent 1.4.0 2019-12-09T21:35:09.719 INFO:teuthology.orchestra.run.smithi067.stdout:Adding gevent 1.4.0 to easy-install.pth file 2019-12-09T21:35:09.720 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.720 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.721 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for bunch==1.0.1 2019-12-09T21:35:09.721 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: bunch 1.0.1 2019-12-09T21:35:09.722 INFO:teuthology.orchestra.run.smithi067.stdout:Adding bunch 1.0.1 to easy-install.pth file 2019-12-09T21:35:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.722 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.722 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for PyYAML==5.2 2019-12-09T21:35:09.722 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: PyYAML 5.2 2019-12-09T21:35:09.722 INFO:teuthology.orchestra.run.smithi067.stdout:Adding PyYAML 5.2 to easy-install.pth file 2019-12-09T21:35:09.722 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.723 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.724 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for boto3==1.10.35 2019-12-09T21:35:09.724 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: boto3 1.10.35 2019-12-09T21:35:09.724 INFO:teuthology.orchestra.run.smithi067.stdout:Adding boto3 1.10.35 to easy-install.pth file 2019-12-09T21:35:09.725 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.725 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.725 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for boto==2.49.0 2019-12-09T21:35:09.725 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: boto 2.49.0 2019-12-09T21:35:09.725 INFO:teuthology.orchestra.run.smithi067.stdout:Adding boto 2.49.0 to easy-install.pth file 2019-12-09T21:35:09.725 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.726 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.726 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for six==1.13.0 2019-12-09T21:35:09.726 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: six 1.13.0 2019-12-09T21:35:09.726 INFO:teuthology.orchestra.run.smithi067.stdout:Adding six 1.13.0 to easy-install.pth file 2019-12-09T21:35:09.731 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.732 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.732 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for greenlet==0.4.15 2019-12-09T21:35:09.732 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: greenlet 0.4.15 2019-12-09T21:35:09.732 INFO:teuthology.orchestra.run.smithi067.stdout:Adding greenlet 0.4.15 to easy-install.pth file 2019-12-09T21:35:09.733 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.733 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.733 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for s3transfer==0.2.1 2019-12-09T21:35:09.733 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: s3transfer 0.2.1 2019-12-09T21:35:09.734 INFO:teuthology.orchestra.run.smithi067.stdout:Adding s3transfer 0.2.1 to easy-install.pth file 2019-12-09T21:35:09.734 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.734 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.734 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for botocore==1.13.35 2019-12-09T21:35:09.735 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: botocore 1.13.35 2019-12-09T21:35:09.735 INFO:teuthology.orchestra.run.smithi067.stdout:Adding botocore 1.13.35 to easy-install.pth file 2019-12-09T21:35:09.735 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.735 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.735 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for jmespath==0.9.4 2019-12-09T21:35:09.736 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: jmespath 0.9.4 2019-12-09T21:35:09.736 INFO:teuthology.orchestra.run.smithi067.stdout:Adding jmespath 0.9.4 to easy-install.pth file 2019-12-09T21:35:09.736 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.736 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.737 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for futures==3.3.0 2019-12-09T21:35:09.737 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: futures 3.3.0 2019-12-09T21:35:09.737 INFO:teuthology.orchestra.run.smithi067.stdout:Adding futures 3.3.0 to easy-install.pth file 2019-12-09T21:35:09.737 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.737 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.738 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for urllib3==1.25.7 2019-12-09T21:35:09.738 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: urllib3 1.25.7 2019-12-09T21:35:09.738 INFO:teuthology.orchestra.run.smithi067.stdout:Adding urllib3 1.25.7 to easy-install.pth file 2019-12-09T21:35:09.738 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.739 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.739 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for python-dateutil==2.8.0 2019-12-09T21:35:09.739 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: python-dateutil 2.8.0 2019-12-09T21:35:09.739 INFO:teuthology.orchestra.run.smithi067.stdout:Adding python-dateutil 2.8.0 to easy-install.pth file 2019-12-09T21:35:09.739 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.740 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.741 INFO:teuthology.orchestra.run.smithi067.stdout:Searching for docutils==0.15.2 2019-12-09T21:35:09.742 INFO:teuthology.orchestra.run.smithi067.stdout:Best match: docutils 0.15.2 2019-12-09T21:35:09.742 INFO:teuthology.orchestra.run.smithi067.stdout:Adding docutils 0.15.2 to easy-install.pth file 2019-12-09T21:35:09.742 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-09T21:35:09.742 INFO:teuthology.orchestra.run.smithi067.stdout:Using /home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages 2019-12-09T21:35:09.743 INFO:teuthology.orchestra.run.smithi067.stdout:Finished processing dependencies for s3tests==0.0.1 2019-12-09T21:35:09.745 INFO:teuthology.orchestra.run.smithi067:> cat > /home/ubuntu/cephtest/archive/s3-tests.client.0.conf 2019-12-09T21:35:09.805 INFO:tasks.s3tests:Configuring boto... 2019-12-09T21:35:09.805 INFO:teuthology.orchestra.run.smithi067:> cat > /home/ubuntu/cephtest/boto.cfg 2019-12-09T21:35:09.907 INFO:teuthology.orchestra.run.smithi067:s3 tests against rgw> S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!fails_with_subdomain' 2019-12-09T21:35:15.808 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_none ... ok 2019-12-09T21:35:17.443 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_empty ... ok 2019-12-09T21:35:21.465 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_date_and_amz_date ... ok 2019-12-09T21:35:24.351 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_amz_date_and_no_date ... ok 2019-12-09T21:35:25.553 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_none ... ok 2019-12-09T21:35:26.992 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_contentlength_none ... ok 2019-12-09T21:35:31.333 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_acl_create_contentlength_none ... ok 2019-12-09T21:35:33.372 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_contentlength_none ... ok 2019-12-09T21:35:34.182 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_empty ... ok 2019-12-09T21:35:34.330 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_none ... ok 2019-12-09T21:35:36.881 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws2 ... ok 2019-12-09T21:35:38.789 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws2 ... ok 2019-12-09T21:35:41.172 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws2 ... ok 2019-12-09T21:35:42.364 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws2 ... ok 2019-12-09T21:35:42.388 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws2 ... ok 2019-12-09T21:35:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws2 ... ok 2019-12-09T21:35:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws4 ... SKIP 2019-12-09T21:35:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_contentlength_mismatch_below_aws4 ... SKIP 2019-12-09T21:35:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_incorrect_aws4 ... SKIP 2019-12-09T21:35:42.447 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_authorization_invalid_aws4 ... SKIP 2019-12-09T21:35:42.448 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_empty_aws4 ... SKIP 2019-12-09T21:35:42.448 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_ua_none_aws4 ... SKIP 2019-12-09T21:35:42.448 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_date_invalid_aws4 ... SKIP 2019-12-09T21:35:42.448 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_invalid_aws4 ... SKIP 2019-12-09T21:35:42.448 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_date_empty_aws4 ... SKIP 2019-12-09T21:35:42.448 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_empty_aws4 ... SKIP 2019-12-09T21:35:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_date_none_aws4 ... SKIP 2019-12-09T21:35:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_none_aws4 ... SKIP 2019-12-09T21:35:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_today_aws4 ... SKIP 2019-12-09T21:35:42.449 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_today_aws4 ... SKIP 2019-12-09T21:35:42.450 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_today_aws4 ... SKIP 2019-12-09T21:35:42.450 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_today_aws4 ... SKIP 2019-12-09T21:35:42.450 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_date_before_epoch_aws4 ... SKIP 2019-12-09T21:35:42.450 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_before_epoch_aws4 ... SKIP 2019-12-09T21:35:42.451 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_date_after_end_aws4 ... SKIP 2019-12-09T21:35:42.451 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_bad_amz_date_after_end_aws4 ... SKIP 2019-12-09T21:35:42.451 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_custom_header_aws4 ... SKIP 2019-12-09T21:35:42.451 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_object_create_missing_signed_header_aws4 ... SKIP 2019-12-09T21:35:42.455 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_authorization_invalid_aws4 ... SKIP 2019-12-09T21:35:42.455 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_empty_aws4 ... SKIP 2019-12-09T21:35:42.455 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_ua_none_aws4 ... SKIP 2019-12-09T21:35:42.455 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_invalid_aws4 ... SKIP 2019-12-09T21:35:42.455 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_invalid_aws4 ... SKIP 2019-12-09T21:35:42.455 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_empty_aws4 ... SKIP 2019-12-09T21:35:42.456 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_empty_aws4 ... SKIP 2019-12-09T21:35:42.456 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_none_aws4 ... SKIP 2019-12-09T21:35:42.456 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_none_aws4 ... SKIP 2019-12-09T21:35:42.457 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_today_aws4 ... SKIP 2019-12-09T21:35:42.457 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_today_aws4 ... SKIP 2019-12-09T21:35:42.457 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_after_today_aws4 ... SKIP 2019-12-09T21:35:42.457 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_after_today_aws4 ... SKIP 2019-12-09T21:35:42.458 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws4 ... SKIP 2019-12-09T21:35:42.458 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_headers.test_bucket_create_bad_amz_date_before_epoch_aws4 ... SKIP 2019-12-09T21:35:45.927 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_versioning_obj_read_not_exist_null ... ok 2019-12-09T21:35:45.928 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_lifecycle_transition ... SKIP 2019-12-09T21:35:45.928 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_lifecycle_transition_single_rule_multi_trans ... SKIP 2019-12-09T21:35:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_lifecycle_set_noncurrent_transition ... SKIP 2019-12-09T21:35:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_object_storage_class ... SKIP 2019-12-09T21:35:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_object_storage_class_multipart ... SKIP 2019-12-09T21:35:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_object_modify_storage_class ... SKIP 2019-12-09T21:35:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_object_modify_storage_class_multipart ... SKIP 2019-12-09T21:35:45.929 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_object_storage_class_copy ... SKIP 2019-12-09T21:35:45.930 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_object_storage_class_copy_multipart ... SKIP 2019-12-09T21:35:49.887 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_atomic_dual_conditional_write_1mb ... ok 2019-12-09T21:35:55.714 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_atomic_write_bucket_gone ... ok 2019-12-09T21:36:03.501 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_1 ... ok 2019-12-09T21:36:06.172 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_encryption_sse_c_multipart_invalid_chunks_2 ... ok 2019-12-09T21:36:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_bucket_policy_set_condition_operator_end_with_IfExists ... ok 2019-12-09T21:36:12.352 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_enc ... ok 2019-12-09T21:36:15.355 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3.test_bucket_policy_put_obj_request_obj_tag ... ok 2019-12-09T21:36:20.128 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.check_can_test_website ... SKIP 2019-12-09T21:36:20.128 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_nonexistant_bucket_rgw ... SKIP 2019-12-09T21:36:20.128 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_public_index ... SKIP 2019-12-09T21:36:20.128 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_public_index ... SKIP 2019-12-09T21:36:20.128 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty ... SKIP 2019-12-09T21:36:20.129 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty ... SKIP 2019-12-09T21:36:20.129 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index ... SKIP 2019-12-09T21:36:20.129 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index ... SKIP 2019-12-09T21:36:20.129 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_missingerrordoc ... SKIP 2019-12-09T21:36:20.129 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_missingerrordoc ... SKIP 2019-12-09T21:36:20.129 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_missingerrordoc ... SKIP 2019-12-09T21:36:20.130 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_missingerrordoc ... SKIP 2019-12-09T21:36:20.133 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_blockederrordoc ... SKIP 2019-12-09T21:36:20.134 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_blockederrordoc ... SKIP 2019-12-09T21:36:20.134 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_blockederrordoc ... SKIP 2019-12-09T21:36:20.134 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_blockederrordoc ... SKIP 2019-12-09T21:36:20.134 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_empty_gooderrordoc ... SKIP 2019-12-09T21:36:20.134 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_empty_gooderrordoc ... SKIP 2019-12-09T21:36:20.134 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_public_bucket_list_private_index_gooderrordoc ... SKIP 2019-12-09T21:36:20.134 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_private_bucket_list_private_index_gooderrordoc ... SKIP 2019-12-09T21:36:20.135 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_base ... SKIP 2019-12-09T21:36:20.135 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path ... SKIP 2019-12-09T21:36:20.135 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_bucket_private_redirectall_path_upgrade ... SKIP 2019-12-09T21:36:20.135 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_xredirect_nonwebsite ... SKIP 2019-12-09T21:36:20.135 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_relative ... SKIP 2019-12-09T21:36:20.135 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_xredirect_public_abs ... SKIP 2019-12-09T21:36:20.136 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_relative ... SKIP 2019-12-09T21:36:20.136 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_s3_website.test_website_xredirect_private_abs ... SKIP 2019-12-09T21:36:20.136 INFO:teuthology.orchestra.run.smithi067.stderr:SKIP 2019-12-09T21:36:20.143 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.functional.test_utils.test_generate ... ok 2019-12-09T21:36:40.640 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:03:01.706 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:36:40.643 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:03:01.710 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:36:40.653 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:03:01.720 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:36:40.655 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:03:01.722 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:36:40.658 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:03:01.816 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:36:40.664 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:03:01.822 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:36:40.799 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:03:01.866 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:36:40.826 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:03:01.893 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:36:40.867 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:03:02.024 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:37:46.495 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_load_graph ... ok 2019-12-09T21:37:46.496 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_descend_leaf_node ... ok 2019-12-09T21:37:46.496 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_descend_node ... ok 2019-12-09T21:37:46.497 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_descend_bad_node ... ok 2019-12-09T21:37:46.497 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_descend_nonexistant_child ... ok 2019-12-09T21:37:46.497 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_printable ... ok 2019-12-09T21:37:46.498 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_binary ... ok 2019-12-09T21:37:46.854 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_printable_no_whitespace ... ok 2019-12-09T21:37:47.011 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_binary_no_whitespace ... ok 2019-12-09T21:37:47.361 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_no_args ... ok 2019-12-09T21:37:47.384 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_no_charset ... ok 2019-12-09T21:37:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_exact_length ... ok 2019-12-09T21:37:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_bad_charset ... ok 2019-12-09T21:37:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_random_missing_length ... ok 2019-12-09T21:37:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_assemble_decision ... ok 2019-12-09T21:37:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_escape ... ok 2019-12-09T21:37:47.404 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_indirect ... ok 2019-12-09T21:37:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_indirect_double ... ok 2019-12-09T21:37:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive ... ok 2019-12-09T21:37:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive_mutual ... ok 2019-12-09T21:37:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_recursive_not_too_eager ... ok 2019-12-09T21:37:47.405 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_make_choice_unweighted_with_space ... ok 2019-12-09T21:37:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_choices ... ok 2019-12-09T21:37:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_null_choices ... ok 2019-12-09T21:37:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_null_choices ... ok 2019-12-09T21:37:47.412 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_null_child ... ok 2019-12-09T21:37:47.413 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_set ... ok 2019-12-09T21:37:47.414 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_header_presence ... ok 2019-12-09T21:37:47.414 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_duplicate_header ... ok 2019-12-09T21:37:47.414 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_headers ... ok 2019-12-09T21:37:47.438 INFO:teuthology.orchestra.run.smithi067.stderr:test_realistic.TestFileValidator.test_new_file_is_valid ... ok 2019-12-09T21:37:47.464 INFO:teuthology.orchestra.run.smithi067.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_on_several_calls ... ok 2019-12-09T21:37:47.465 INFO:teuthology.orchestra.run.smithi067.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_when_size_is_1 ... ok 2019-12-09T21:37:47.612 INFO:teuthology.orchestra.run.smithi067.stderr:test_realistic.TestFiles.test_random_file_valid ... ok 2019-12-09T21:37:51.113 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_short ... ok 2019-12-09T21:38:01.700 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_bad ... ok 2019-12-09T21:38:03.812 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_empty ... ok 2019-12-09T21:38:08.070 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_none ... ok 2019-12-09T21:38:12.983 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_mismatch ... ok 2019-12-09T21:38:17.522 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_empty ... ok 2019-12-09T21:38:21.810 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_expect_none ... ok 2019-12-09T21:38:23.280 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contentlength_negative ... ok 2019-12-09T21:38:28.673 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_invalid ... ok 2019-12-09T21:38:32.054 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_empty ... ok 2019-12-09T21:38:36.393 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_contenttype_none ... ok 2019-12-09T21:38:37.650 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_put_bad_canned_acl ... ok 2019-12-09T21:38:38.671 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_mismatch ... ok 2019-12-09T21:38:40.630 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_expect_empty ... ok 2019-12-09T21:38:40.723 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_contentlength_negative ... ok 2019-12-09T21:38:43.145 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_md5_invalid_garbage_aws2 ... ok 2019-12-09T21:38:47.938 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_empty_aws2 ... ok 2019-12-09T21:38:53.393 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_ua_none_aws2 ... ok 2019-12-09T21:38:55.091 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_invalid_aws2 ... ok 2019-12-09T21:38:56.988 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_empty_aws2 ... ok 2019-12-09T21:38:59.160 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_today_aws2 ... ok 2019-12-09T21:39:01.548 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_before_epoch_aws2 ... ok 2019-12-09T21:39:03.379 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_object_create_bad_date_after_end_aws2 ... ok 2019-12-09T21:39:05.943 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_empty_aws2 ... ok 2019-12-09T21:39:07.951 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_ua_none_aws2 ... ok 2019-12-09T21:39:08.261 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_invalid_aws2 ... ok 2019-12-09T21:39:08.281 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_empty_aws2 ... ok 2019-12-09T21:39:08.313 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_today_aws2 ... ok 2019-12-09T21:39:08.344 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_after_today_aws2 ... ok 2019-12-09T21:39:08.469 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_headers.test_bucket_create_bad_date_before_epoch_aws2 ... ok 2019-12-09T21:39:10.401 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_empty ... ok 2019-12-09T21:39:15.761 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_distinct ... ok 2019-12-09T21:39:22.176 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_many ... ok 2019-12-09T21:39:26.603 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_many ... ok 2019-12-09T21:39:34.267 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_basic_key_count ... ok 2019-12-09T21:39:40.926 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_basic ... ok 2019-12-09T21:39:47.343 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_basic ... ok 2019-12-09T21:39:54.595 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_encoding_basic ... ok 2019-12-09T21:40:00.537 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_encoding_basic ... ok 2019-12-09T21:40:14.061 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix ... ok 2019-12-09T21:40:27.238 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix ... ok 2019-12-09T21:40:31.340 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_ends_with_delimiter ... ok 2019-12-09T21:40:35.301 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_ends_with_delimiter ... ok 2019-12-09T21:40:41.676 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_alt ... ok 2019-12-09T21:40:47.118 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_alt ... ok 2019-12-09T21:41:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_prefix_underscore ... ok 2019-12-09T21:41:13.881 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_prefix_underscore ... ok 2019-12-09T21:41:16.956 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:07:38.022 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:16.961 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:07:38.025 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:16.996 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:07:38.061 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:17.018 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:07:38.084 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:17.045 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:07:38.203 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:17.052 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:07:38.209 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:17.095 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:07:38.162 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:17.169 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:07:38.236 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:17.184 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:07:38.336 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_percentage ... ok 2019-12-09T21:41:26.533 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_percentage ... ok 2019-12-09T21:41:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_whitespace ... ok 2019-12-09T21:41:38.797 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_whitespace ... ok 2019-12-09T21:41:44.514 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_dot ... ok 2019-12-09T21:41:51.197 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_dot ... ok 2019-12-09T21:41:52.113 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:08:13.178 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:52.123 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:08:13.187 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:52.130 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:08:13.197 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:52.135 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:08:13.202 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:52.145 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:08:13.296 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:52.146 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:08:13.301 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:52.171 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:08:13.234 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:52.186 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:08:13.253 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:52.212 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:08:13.369 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:41:57.595 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_unreadable ... ok 2019-12-09T21:42:06.260 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_unreadable ... ok 2019-12-09T21:42:12.444 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_empty ... ok 2019-12-09T21:42:18.230 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_empty ... ok 2019-12-09T21:42:23.892 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_none ... ok 2019-12-09T21:42:29.832 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_none ... ok 2019-12-09T21:42:35.266 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_notempty ... ok 2019-12-09T21:42:40.244 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_defaultempty ... ok 2019-12-09T21:42:46.700 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_fetchowner_empty ... ok 2019-12-09T21:42:54.243 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_exist ... ok 2019-12-09T21:43:00.787 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_delimiter_not_exist ... ok 2019-12-09T21:47:04.399 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:13:25.464 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:47:04.408 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:13:25.474 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:47:04.420 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:13:25.577 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:47:04.433 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:13:25.499 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:47:04.437 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:13:25.595 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:47:04.440 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:13:25.506 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:47:04.629 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:13:25.695 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:47:04.629 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:13:25.786 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:47:04.631 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:13:25.697 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:49:38.435 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:15:59.501 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:49:38.489 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:15:59.555 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:49:38.494 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:15:59.560 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:49:38.497 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:15:59.563 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:49:38.521 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:15:59.678 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:49:38.528 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:15:59.685 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:49:38.591 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:15:59.656 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:49:38.612 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:15:59.678 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:49:38.664 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:15:59.821 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:52:15.015 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:18:36.081 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:52:15.039 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:18:36.185 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:52:15.039 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:18:36.189 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:52:15.039 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:18:36.090 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:52:15.039 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:18:36.096 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:52:15.040 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:18:36.099 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:52:15.133 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:18:36.290 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:52:15.134 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:18:36.200 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:52:15.150 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:18:36.216 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:24.872 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:23:45.938 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:24.889 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:23:46.042 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:24.890 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:23:46.045 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:24.890 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:23:45.947 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:24.890 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:23:45.952 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:24.891 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:23:45.957 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:25.081 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:23:46.148 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:25.086 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:23:46.243 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:25.128 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:23:46.195 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:59.988 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:24:21.055 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:57:59.992 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:24:21.059 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:58:00.020 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:24:21.087 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:58:00.040 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:24:21.108 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T21:58:00.047 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:24:21.114 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:58:00.077 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:24:21.234 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:58:00.077 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:24:21.144 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T21:58:00.086 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:24:21.243 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:58:00.145 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:24:21.302 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T21:59:57.191 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_delimiter_not_skip_special ... ok 2019-12-09T22:00:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_basic ... ok 2019-12-09T22:00:07.360 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_basic ... ok 2019-12-09T22:00:13.434 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_alt ... ok 2019-12-09T22:00:19.553 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_alt ... ok 2019-12-09T22:00:26.295 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_empty ... ok 2019-12-09T22:00:31.385 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_empty ... ok 2019-12-09T22:00:37.017 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_none ... ok 2019-12-09T22:00:43.255 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_none ... ok 2019-12-09T22:00:47.906 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_not_exist ... ok 2019-12-09T22:00:53.213 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_not_exist ... ok 2019-12-09T22:00:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_unreadable ... ok 2019-12-09T22:01:03.741 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_unreadable ... ok 2019-12-09T22:01:10.169 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_basic ... ok 2019-12-09T22:01:18.562 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_basic ... ok 2019-12-09T22:01:26.340 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_alt ... ok 2019-12-09T22:01:33.539 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_alt ... ok 2019-12-09T22:01:41.486 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_not_exist ... ok 2019-12-09T22:01:49.582 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_not_exist ... ok 2019-12-09T22:01:55.657 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_delimiter_not_exist ... ok 2019-12-09T22:02:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_delimiter_not_exist ... ok 2019-12-09T22:02:09.041 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_prefix_delimiter_prefix_delimiter_not_exist ... ok 2019-12-09T22:02:16.962 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_prefix_delimiter_prefix_delimiter_not_exist ... ok 2019-12-09T22:02:23.824 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_one ... ok 2019-12-09T22:02:29.986 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_one ... ok 2019-12-09T22:02:35.484 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:28:56.551 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:02:35.492 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:28:56.559 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:02:35.504 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:28:56.571 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:02:35.505 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:28:56.662 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:02:35.508 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:28:56.666 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:02:35.509 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:28:56.576 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:02:35.678 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:28:56.835 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:02:35.729 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:28:56.797 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:02:35.734 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:28:56.802 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:02:36.153 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_zero ... ok 2019-12-09T22:02:42.119 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_zero ... ok 2019-12-09T22:02:49.864 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_none ... ok 2019-12-09T22:02:56.049 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_maxkeys_none ... ok 2019-12-09T22:03:33.606 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_unordered ... ok 2019-12-09T22:04:05.907 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_unordered ... ok 2019-12-09T22:04:11.804 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_maxkeys_invalid ... ok 2019-12-09T22:04:18.740 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_none ... ok 2019-12-09T22:04:27.031 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_empty ... ok 2019-12-09T22:04:33.222 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_continuationtoken_empty ... ok 2019-12-09T22:04:39.330 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_unreadable ... ok 2019-12-09T22:04:47.520 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_unreadable ... ok 2019-12-09T22:04:54.401 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_not_in_list ... ok 2019-12-09T22:05:02.691 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_not_in_list ... ok 2019-12-09T22:05:11.409 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_marker_after_list ... ok 2019-12-09T22:05:18.472 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_startafter_after_list ... ok 2019-12-09T22:05:25.926 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data ... ok 2019-12-09T22:05:45.149 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_return_data_versioning ... ok 2019-12-09T22:05:47.617 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous ... ok 2019-12-09T22:05:50.597 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous ... ok 2019-12-09T22:05:51.939 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_objects_anonymous_fail ... ok 2019-12-09T22:05:53.998 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_listv2_objects_anonymous_fail ... ok 2019-12-09T22:05:54.906 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_notexist ... ok 2019-12-09T22:05:55.176 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_notexist ... ok 2019-12-09T22:05:55.476 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_notexist ... ok 2019-12-09T22:05:59.408 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_delete_nonempty ... ok 2019-12-09T22:06:06.727 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_concurrent_set_canned_acl ... ok 2019-12-09T22:06:07.026 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_write_to_nonexist_bucket ... ok 2019-12-09T22:06:12.928 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_delete ... ok 2019-12-09T22:06:14.873 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_read_notexist ... ok 2019-12-09T22:06:16.679 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_requestid_matches_header_on_error ... ok 2019-12-09T22:06:21.738 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:32:42.804 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:06:21.782 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:32:42.849 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:06:21.783 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:32:42.849 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:06:21.803 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:32:42.870 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:06:21.839 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:32:42.996 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:06:21.845 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:32:43.003 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:06:21.903 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:32:42.970 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:06:21.904 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:32:42.971 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:06:21.984 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:32:43.141 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:06:24.835 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multi_object_delete ... ok 2019-12-09T22:06:36.196 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multi_objectv2_delete ... ok 2019-12-09T22:06:38.875 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_head_zero_bytes ... ok 2019-12-09T22:06:40.316 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_write_check_etag ... ok 2019-12-09T22:06:43.711 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_write_cache_control ... ok 2019-12-09T22:06:46.157 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_write_expires ... ok 2019-12-09T22:06:53.176 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_write_read_update_read_delete ... ok 2019-12-09T22:06:55.738 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_good ... ok 2019-12-09T22:06:58.829 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_none_to_empty ... ok 2019-12-09T22:07:03.958 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_metadata_overwrite_to_empty ... ok 2019-12-09T22:07:06.924 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_set_get_unicode_metadata ... ok 2019-12-09T22:07:10.487 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_metadata_replaced_on_put ... ok 2019-12-09T22:07:13.477 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_write_file ... ok 2019-12-09T22:07:17.236 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_anonymous_request ... ok 2019-12-09T22:07:22.768 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request ... ok 2019-12-09T22:07:26.493 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_no_content_type ... ok 2019-12-09T22:07:28.458 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_authenticated_request_bad_access_key ... ok 2019-12-09T22:07:32.212 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_success_code ... ok 2019-12-09T22:07:37.049 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_invalid_success_code ... ok 2019-12-09T22:07:42.928 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_larger_than_chunk ... ok 2019-12-09T22:07:46.402 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_set_key_from_filename ... ok 2019-12-09T22:07:47.026 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:34:08.093 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:07:47.029 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:34:08.096 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:07:47.041 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:34:08.108 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:07:47.046 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:34:08.113 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:07:47.052 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:34:08.209 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:07:47.061 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:34:08.218 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:07:47.118 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:34:08.184 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:07:47.128 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:34:08.194 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:07:47.194 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:34:08.351 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:07:48.190 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_ignored_header ... ok 2019-12-09T22:07:51.271 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_case_insensitive_condition_fields ... ok 2019-12-09T22:07:56.725 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_escaped_field_values ... ok 2019-12-09T22:08:00.846 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_success_redirect_action ... ok 2019-12-09T22:08:03.815 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_signature ... ok 2019-12-09T22:08:06.726 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_access_key ... ok 2019-12-09T22:08:09.823 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_date_format ... ok 2019-12-09T22:08:12.712 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_no_key_specified ... ok 2019-12-09T22:08:15.291 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_signature ... ok 2019-12-09T22:08:18.036 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_policy_condition ... ok 2019-12-09T22:08:22.020 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_user_specified_header ... ok 2019-12-09T22:08:23.799 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_request_missing_policy_specified_field ... ok 2019-12-09T22:08:26.837 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_condition_is_case_sensitive ... ok 2019-12-09T22:08:30.058 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_expires_is_case_sensitive ... ok 2019-12-09T22:08:32.371 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_expired_policy ... ok 2019-12-09T22:08:35.209 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_request_field_value ... ok 2019-12-09T22:08:37.288 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_expires_condition ... ok 2019-12-09T22:08:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_conditions_list ... ok 2019-12-09T22:08:43.505 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_limit_exceeded ... ok 2019-12-09T22:08:46.462 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_missing_content_length_argument ... ok 2019-12-09T22:08:48.841 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_invalid_content_length_argument ... ok 2019-12-09T22:08:51.401 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_upload_size_below_minimum ... ok 2019-12-09T22:08:53.475 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_empty_conditions ... ok 2019-12-09T22:08:57.023 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_good ... ok 2019-12-09T22:09:00.943 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmatch_failed ... ok 2019-12-09T22:09:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_good ... ok 2019-12-09T22:09:08.839 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifnonematch_failed ... ok 2019-12-09T22:09:12.182 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_good ... ok 2019-12-09T22:09:17.277 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifmodifiedsince_failed ... ok 2019-12-09T22:09:20.736 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_good ... ok 2019-12-09T22:09:24.365 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_object_ifunmodifiedsince_failed ... ok 2019-12-09T22:09:28.787 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_good ... ok 2019-12-09T22:09:33.705 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_failed ... ok 2019-12-09T22:09:38.965 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_overwrite_existed_good ... ok 2019-12-09T22:09:42.073 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifmatch_nonexisted_failed ... ok 2019-12-09T22:09:48.342 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_good ... ok 2019-12-09T22:09:52.409 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_failed ... ok 2019-12-09T22:09:57.822 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_nonexisted_good ... ok 2019-12-09T22:10:02.760 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_object_ifnonmatch_overwrite_existed_failed ... ok 2019-12-09T22:10:06.740 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get ... ok 2019-12-09T22:10:13.796 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_gone ... ok 2019-12-09T22:10:22.570 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_delete_key_bucket_gone ... ok 2019-12-09T22:10:25.563 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_gone ... ok 2019-12-09T22:10:27.647 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_head ... ok 2019-12-09T22:10:37.912 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_head_extended ... ok 2019-12-09T22:10:40.223 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_bucket_acl ... ok 2019-12-09T22:10:43.457 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_object_acl ... ok 2019-12-09T22:10:47.329 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated ... ok 2019-12-09T22:10:50.402 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_response_headers ... ok 2019-12-09T22:10:54.089 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_acl ... ok 2019-12-09T22:10:57.423 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_acl ... ok 2019-12-09T22:11:04.634 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_bucket_gone ... ok 2019-12-09T22:11:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_authenticated_object_gone ... ok 2019-12-09T22:11:11.702 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_not_expired ... ok 2019-12-09T22:11:14.921 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_range_zero ... ok 2019-12-09T22:11:18.337 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_max_range ... ok 2019-12-09T22:11:22.462 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_get_x_amz_expires_out_positive_range ... ok 2019-12-09T22:11:26.282 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put ... ok 2019-12-09T22:11:31.276 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_anon_put_write_access ... ok 2019-12-09T22:11:34.625 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_put_authenticated ... ok 2019-12-09T22:11:38.503 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_raw_put_authenticated_expired ... ok 2019-12-09T22:11:39.427 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_one ... ok 2019-12-09T22:11:40.109 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_short_two ... ok 2019-12-09T22:11:40.535 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_bad_ip ... ok 2019-12-09T22:11:41.214 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_dns_underscore ... ok 2019-12-09T22:11:44.431 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists ... ok 2019-12-09T22:11:46.562 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_get_location ... ok 2019-12-09T22:11:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_exists_nonowner ... ok 2019-12-09T22:11:55.799 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_default ... ok 2019-12-09T22:11:58.745 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_during_create ... ok 2019-12-09T22:12:02.442 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned ... ok 2019-12-09T22:12:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_publicreadwrite ... ok 2019-12-09T22:12:06.879 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_authenticatedread ... ok 2019-12-09T22:12:11.752 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_default ... ok 2019-12-09T22:12:15.155 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_during_create ... ok 2019-12-09T22:12:21.313 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned ... ok 2019-12-09T22:12:25.945 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_publicreadwrite ... ok 2019-12-09T22:12:29.769 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_authenticatedread ... ok 2019-12-09T22:12:33.305 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerread ... ok 2019-12-09T22:12:39.069 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_canned_bucketownerfullcontrol ... ok 2019-12-09T22:12:44.925 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_owner ... ok 2019-12-09T22:12:50.259 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_full_control_verify_attributes ... ok 2019-12-09T22:12:54.263 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_canned_private_to_private ... ok 2019-12-09T22:12:59.811 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:39:20.877 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:12:59.821 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:39:20.887 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:12:59.830 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:39:20.896 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:12:59.836 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:39:20.994 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:12:59.837 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:39:20.903 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:12:59.852 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:39:21.010 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:12:59.874 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl ... ok 2019-12-09T22:13:00.095 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:39:21.161 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:13:00.098 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:39:21.164 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:13:00.110 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:39:21.268 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:13:04.810 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_write ... ok 2019-12-09T22:13:10.021 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_writeacp ... ok 2019-12-09T22:13:16.023 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_read ... ok 2019-12-09T22:13:19.935 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_acl_readacp ... ok 2019-12-09T22:13:26.101 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_fullcontrol ... ok 2019-12-09T22:13:30.081 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_read ... ok 2019-12-09T22:13:34.146 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_readacp ... ok 2019-12-09T22:13:36.976 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_write ... ok 2019-12-09T22:13:41.402 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_userid_writeacp ... ok 2019-12-09T22:13:43.743 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_nonexist_user ... ok 2019-12-09T22:13:50.921 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_no_grants ... ok 2019-12-09T22:13:54.905 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_header_acl_grants ... ok 2019-12-09T22:13:59.116 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_header_acl_grants ... ok 2019-12-09T22:14:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email ... ok 2019-12-09T22:14:06.704 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_grant_email_notexist ... ok 2019-12-09T22:14:11.755 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_acl_revoke_all ... ok 2019-12-09T22:14:19.435 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_private ... ok 2019-12-09T22:14:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_private ... ok 2019-12-09T22:14:37.171 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicread ... ok 2019-12-09T22:14:43.190 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:41:04.248 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:14:43.190 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:41:04.254 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:14:43.230 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:41:04.295 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:14:43.249 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:41:04.315 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:14:43.252 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:41:04.411 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:14:43.255 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:41:04.413 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:14:43.261 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:41:04.327 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:14:43.309 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:41:04.374 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:14:43.313 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:41:04.471 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:14:44.444 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicread ... ok 2019-12-09T22:14:53.697 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_object_publicreadwrite ... ok 2019-12-09T22:15:00.602 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_private_objectv2_publicreadwrite ... ok 2019-12-09T22:15:08.537 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:41:29.603 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:15:08.563 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:41:29.628 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:15:08.591 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:41:29.657 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:15:08.599 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:41:29.664 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:15:08.609 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:41:29.675 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:15:08.643 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:41:29.709 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:15:08.652 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:41:29.809 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:15:08.659 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:41:29.816 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:15:08.701 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:41:29.856 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:15:09.845 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_private ... ok 2019-12-09T22:15:18.515 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicread ... ok 2019-12-09T22:15:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicread_object_publicreadwrite ... ok 2019-12-09T22:15:42.952 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_private ... ok 2019-12-09T22:15:51.691 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicread ... ok 2019-12-09T22:16:00.045 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_access_bucket_publicreadwrite_object_publicreadwrite ... ok 2019-12-09T22:16:10.306 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_buckets_create_then_list ... ok 2019-12-09T22:16:10.925 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_anonymous ... ok 2019-12-09T22:16:11.270 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_invalid_auth ... ok 2019-12-09T22:16:11.382 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_list_buckets_bad_auth ... ok 2019-12-09T22:16:19.437 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_alpha ... ok 2019-12-09T22:16:28.653 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_starts_digit ... ok 2019-12-09T22:16:30.412 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_period ... ok 2019-12-09T22:16:31.601 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_naming_good_contains_hyphen ... ok 2019-12-09T22:16:38.478 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_recreate_not_overriding ... ok 2019-12-09T22:17:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_create_special_key_names ... ok 2019-12-09T22:17:17.181 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_list_special_prefix ... ok 2019-12-09T22:17:25.813 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_zero_size ... ok 2019-12-09T22:17:31.200 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_same_bucket ... ok 2019-12-09T22:17:35.789 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_verify_contenttype ... ok 2019-12-09T22:17:42.049 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself ... ok 2019-12-09T22:17:46.569 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_to_itself_with_metadata ... ok 2019-12-09T22:17:53.100 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_diff_bucket ... ok 2019-12-09T22:17:59.253 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_bucket ... ok 2019-12-09T22:18:08.883 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_not_owned_object_bucket ... ok 2019-12-09T22:18:09.159 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:44:30.224 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:18:09.173 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:44:30.235 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:18:09.187 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:44:30.251 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:18:09.187 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:44:30.346 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:18:09.197 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:44:30.354 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:18:09.197 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:44:30.259 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:18:09.290 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:44:30.355 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:18:09.324 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:44:30.389 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:18:09.377 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:44:30.532 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:18:15.588 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_canned_acl ... ok 2019-12-09T22:18:27.914 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_retaining_metadata ... ok 2019-12-09T22:18:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_replacing_metadata ... ok 2019-12-09T22:18:39.519 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_bucket_not_found ... ok 2019-12-09T22:18:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_key_not_found ... ok 2019-12-09T22:19:17.111 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_bucket ... ok 2019-12-09T22:19:27.830 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioned_url_encoding ... ok 2019-12-09T22:21:01.535 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_copy_versioning_multipart_upload ... ok 2019-12-09T22:21:03.792 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_empty ... ok 2019-12-09T22:21:12.060 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_small ... ok 2019-12-09T22:21:28.396 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:21:28.380+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:21:28.897 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_small ... ok 2019-12-09T22:21:34.913 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_invalid_range ... ok 2019-12-09T22:21:49.810 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_without_range ... ok 2019-12-09T22:22:08.069 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:22:08.063+0000 40262700 -1 handle_data failed with exit code -5 2019-12-09T22:22:23.019 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:22:22.996+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:22:37.908 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:22:37.797+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:22:53.735 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:22:53.732+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:22:54.152 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_special_names ... ok 2019-12-09T22:23:05.011 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:49:26.076 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:05.019 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:49:26.083 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:05.076 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:49:26.235 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:05.097 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:49:26.256 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:05.103 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:49:26.167 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:05.106 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:49:26.171 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:05.169 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:49:26.229 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:05.217 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:49:26.376 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:05.262 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:49:26.327 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:22.044 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:49:43.109 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:22.051 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:49:43.116 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:22.061 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:49:43.126 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:22.065 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:49:43.130 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:22.078 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:49:43.237 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:22.087 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:49:43.246 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:22.097 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:49:43.155 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:22.104 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:49:43.169 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:23:22.168 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:49:43.327 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:24:11.977 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload ... ok 2019-12-09T22:25:21.582 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:25:21.463+0000 40262700 -1 handle_data failed with exit code -5 2019-12-09T22:25:25.153 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:25:25.084+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:25:42.457 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:25:42.420+0000 40a63700 -1 handle_data failed with exit code -5 2019-12-09T22:25:47.513 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_versioned ... ok 2019-12-09T22:25:47.551 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:25:47.540+0000 40262700 -1 handle_data failed with exit code -5 2019-12-09T22:27:27.362 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:27:27.305+0000 40262700 -1 handle_data failed with exit code -5 2019-12-09T22:27:56.770 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:27:56.760+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:28:31.901 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:28:31.884+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:28:35.293 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:54:56.358 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:28:35.296 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:54:56.361 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:28:35.307 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:54:56.372 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:28:35.312 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:54:56.377 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:28:35.318 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:54:56.477 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:28:35.328 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:54:56.487 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:28:35.496 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:54:56.561 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:28:35.528 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:54:56.686 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:28:35.535 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:54:56.600 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:28:54.078 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:28:53.979+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:29:40.365 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:29:40.274+0000 40262700 -1 handle_data failed with exit code -5 2019-12-09T22:30:14.166 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:30:14.042+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:31:10.178 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:31:10.146+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:31:26.469 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:57:47.534 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:31:26.493 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:57:47.557 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:31:26.537 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:57:47.598 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:31:26.537 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:57:47.599 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:31:26.569 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:57:47.727 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:31:26.572 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:57:47.730 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:31:26.611 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:01:57:47.676 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:31:26.636 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:01:57:47.702 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:31:26.681 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:01:57:47.833 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:31:37.847 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-09T22:31:37.843+0000 3fa61700 -1 handle_data failed with exit code -5 2019-12-09T22:31:41.654 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_resend_part ... ok 2019-12-09T22:32:54.017 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_multiple_sizes ... ok 2019-12-09T22:33:44.508 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:00:05.573 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:33:44.516 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:00:05.582 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:33:44.520 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:00:05.586 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:33:44.524 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:00:05.590 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:33:44.536 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:00:05.694 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:33:44.549 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:00:05.707 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:33:44.618 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:00:05.683 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:33:44.675 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:00:05.740 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:33:44.737 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:00:05.895 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:34:45.506 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_copy_multiple_sizes ... ok 2019-12-09T22:35:11.095 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_size_too_small ... ok 2019-12-09T22:35:30.988 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_contents ... ok 2019-12-09T22:35:47.214 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_overwrite_existing_object ... ok 2019-12-09T22:35:58.297 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload ... ok 2019-12-09T22:36:00.372 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_abort_multipart_upload_not_found ... ok 2019-12-09T22:36:23.061 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload ... ok 2019-12-09T22:36:36.751 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_list_multipart_upload_encoding ... ok 2019-12-09T22:36:42.871 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_missing_part ... ok 2019-12-09T22:36:48.536 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_upload_incorrect_etag ... ok 2019-12-09T22:36:52.428 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_100_continue ... ok 2019-12-09T22:36:56.282 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_set_cors ... ok 2019-12-09T22:37:15.204 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_response ... ok 2019-12-09T22:37:24.112 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_cors_origin_wildcard ... ok 2019-12-09T22:37:30.289 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_cors_header_option ... ok 2019-12-09T22:37:37.430 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_1mb ... ok 2019-12-09T22:37:47.156 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_4mb ... ok 2019-12-09T22:38:05.776 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_read_8mb ... ok 2019-12-09T22:38:14.898 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_1mb ... ok 2019-12-09T22:38:25.809 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_4mb ... ok 2019-12-09T22:38:39.138 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_write_8mb ... ok 2019-12-09T22:38:50.044 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_1mb ... ok 2019-12-09T22:38:56.509 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:05:17.576 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:38:56.516 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:05:17.583 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:38:56.529 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:05:17.595 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:38:56.533 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:05:17.600 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:38:56.533 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:05:17.692 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:38:56.542 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:05:17.701 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:38:56.733 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:05:17.799 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:38:56.787 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:05:17.854 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:38:56.834 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:05:17.993 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:07.850 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_4mb ... ok 2019-12-09T22:39:27.005 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_dual_write_8mb ... ok 2019-12-09T22:39:35.851 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_conditional_write_1mb ... ok 2019-12-09T22:39:43.370 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_atomic_multipart_upload_write ... ok 2019-12-09T22:39:48.110 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:06:09.176 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:48.161 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:06:09.227 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:48.162 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:06:09.229 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:48.162 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:06:09.229 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:48.187 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:06:09.345 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:48.189 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:06:09.348 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:48.213 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:06:09.279 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:48.222 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:06:09.287 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:48.248 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:06:09.406 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:39:59.772 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_multipart_resend_first_finishes_last ... ok 2019-12-09T22:40:02.182 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_response_code ... ok 2019-12-09T22:40:12.569 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_ranged_big_request_response_code ... ok 2019-12-09T22:40:15.899 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_skip_leading_bytes_response_code ... ok 2019-12-09T22:40:19.038 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_return_trailing_bytes_response_code ... ok 2019-12-09T22:40:23.153 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_invalid_range ... ok 2019-12-09T22:40:27.604 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_ranged_request_empty_object ... ok 2019-12-09T22:40:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_create_suspend ... ok 2019-12-09T22:44:08.731 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:10:29.798 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:44:08.735 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:10:29.802 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:44:08.741 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:10:29.808 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:44:08.744 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:10:29.902 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:44:08.747 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:10:29.814 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:44:08.748 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:10:29.906 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:44:08.901 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:10:29.968 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:44:08.959 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:10:30.026 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:44:08.962 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:10:30.119 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:44:48.295 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove ... ok 2019-12-09T22:45:39.753 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_read_remove_head ... ok 2019-12-09T22:45:48.203 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_removal ... ok 2019-12-09T22:46:00.157 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite ... ok 2019-12-09T22:46:11.316 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_plain_null_version_overwrite_suspended ... ok 2019-12-09T22:47:42.309 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_suspend_versions ... ok 2019-12-09T22:48:09.969 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:14:31.037 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:48:09.994 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:14:31.062 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:48:10.009 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:14:31.077 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:48:10.035 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:14:31.103 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:48:10.042 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:14:31.200 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:48:10.066 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:14:31.224 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:48:10.110 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:14:31.178 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:48:10.167 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:14:31.235 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:48:10.174 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:14:31.333 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:49:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_all ... ok 2019-12-09T22:49:20.258 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:15:41.326 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:49:20.266 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:15:41.334 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:49:20.271 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:15:41.430 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:49:20.275 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:15:41.343 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:49:20.276 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:15:41.434 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:49:20.277 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:15:41.346 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:49:20.332 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:15:41.400 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:49:20.344 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:15:41.502 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:49:20.356 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:15:41.424 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:54:31.194 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:20:52.262 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:54:31.200 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:20:52.268 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:54:31.213 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:20:52.280 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:54:31.217 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:20:52.284 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:54:31.220 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:20:52.377 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:54:31.225 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:20:52.382 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:54:31.420 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:20:52.578 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:54:31.434 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:20:52.500 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:54:31.455 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:20:52.513 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:55:46.647 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_versions_remove_special_names ... ok 2019-12-09T22:56:06.751 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:22:27.818 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:06.758 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:22:27.825 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:06.776 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:22:27.934 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:06.785 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:22:27.943 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:06.815 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:22:27.882 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:06.816 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:22:27.884 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:06.862 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:22:28.020 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:06.870 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:22:27.937 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:06.899 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:22:27.960 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:31.874 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:22:52.941 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:31.890 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:22:53.048 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:31.891 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:22:52.958 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:31.893 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:22:53.052 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:31.904 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:22:52.971 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:31.909 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:22:52.976 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:31.924 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:22:53.082 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:31.936 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:22:53.004 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:56:31.937 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:22:53.005 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:57:16.718 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_create_overwrite_multipart ... ok 2019-12-09T22:57:54.555 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_obj_list_marker ... ok 2019-12-09T22:58:37.299 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_copy_obj_version ... ok 2019-12-09T22:59:00.499 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete ... ok 2019-12-09T22:59:27.169 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker ... ok 2019-12-09T22:59:32.661 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_multi_object_delete_with_marker_create ... ok 2019-12-09T22:59:41.264 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:26:02.329 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:59:41.409 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:26:02.444 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:59:41.409 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:26:02.448 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:59:41.410 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:26:02.341 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:59:41.410 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:26:02.349 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:59:41.410 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:26:02.355 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:59:41.426 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:26:02.493 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T22:59:41.442 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:26:02.600 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T22:59:41.463 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:26:02.530 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T22:59:49.234 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl ... ok 2019-12-09T23:00:07.516 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioned_object_acl_no_version_specified ... ok 2019-12-09T23:00:15.345 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_concurrent_remove ... Exception in thread Thread-53: 2019-12-09T23:00:15.347 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:15.347 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:15.347 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:15.347 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:15.348 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:15.348 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:15.348 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:15.348 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:15.349 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:15.349 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:15.349 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:15.349 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:15.350 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-51: 2019-12-09T23:00:15.351 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:15.351 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:15.351 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:15.352 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:15.352 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:15.352 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:15.352 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:15.352 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:15.353 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:15.353 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:15.353 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:15.353 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:15.354 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:15.406 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-54: 2019-12-09T23:00:15.406 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:15.406 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:15.406 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:15.407 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:15.407 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:15.407 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:15.407 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:15.407 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:15.408 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:15.408 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:15.408 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:15.408 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:15.409 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-55: 2019-12-09T23:00:15.481 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:15.482 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:15.482 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:15.482 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:15.482 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:15.483 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:15.483 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:15.483 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:15.483 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:15.484 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:15.484 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:15.484 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:15.484 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:23.114 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-63: 2019-12-09T23:00:23.115 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:23.115 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:23.115 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:23.115 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:23.116 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:23.116 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:23.116 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:23.116 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:23.116 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:23.116 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:23.116 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:23.117 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:23.117 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:23.165 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-64: 2019-12-09T23:00:23.166 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:23.166 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:23.166 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:23.166 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:23.167 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:23.167 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:23.167 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:23.167 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:23.168 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:23.168 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:23.168 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:23.168 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:23.168 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-61: 2019-12-09T23:00:23.246 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:23.247 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:23.247 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:23.247 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:23.247 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:23.248 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:23.248 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:23.248 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:23.248 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:23.248 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:23.249 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:23.249 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:23.249 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:33.589 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-73: 2019-12-09T23:00:33.601 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:33.601 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:33.602 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:33.602 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:33.602 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:33.602 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:33.602 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:33.602 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:33.603 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:34.212 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:34.213 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:34.213 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:34.213 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:34.214 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-74: 2019-12-09T23:00:34.214 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:34.214 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:34.214 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:34.215 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:34.215 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:34.215 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:34.215 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:34.215 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:34.215 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:34.215 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:34.216 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:34.216 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:34.216 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:34.216 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-71: 2019-12-09T23:00:34.216 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:34.216 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:34.217 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:34.217 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:34.217 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:34.217 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:34.217 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:34.217 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:34.217 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:34.218 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:34.218 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:34.218 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-72: 2019-12-09T23:00:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:34.218 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:34.219 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:34.219 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:34.219 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:34.219 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:34.219 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:34.219 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:34.219 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:34.220 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:34.220 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:34.220 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:46.053 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-85: 2019-12-09T23:00:46.055 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:46.056 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:46.056 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:46.059 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:46.059 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:46.059 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:46.059 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:46.060 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:46.060 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:46.060 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:46.060 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:46.060 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:46.061 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:46.968 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-83: 2019-12-09T23:00:46.969 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:46.969 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:46.969 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:46.969 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:46.969 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:46.969 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:46.969 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:46.970 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:46.970 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:46.970 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:46.970 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:46.970 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:46.970 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:47.286 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-81: 2019-12-09T23:00:47.287 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:47.287 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:47.287 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:47.287 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:47.288 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:47.288 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:47.288 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:47.288 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:47.289 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:47.289 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:47.289 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:47.289 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:47.289 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-82: 2019-12-09T23:00:47.324 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:47.324 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:47.324 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:47.325 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:47.325 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:47.325 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:47.325 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:47.325 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:47.325 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:47.325 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:47.326 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:47.326 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:47.326 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:56.393 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-93: 2019-12-09T23:00:56.394 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:56.394 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:56.395 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:56.395 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:56.395 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:56.395 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:56.395 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:56.395 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:56.396 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:56.396 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:56.396 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:56.396 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-95: 2019-12-09T23:00:56.396 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:56.397 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:56.397 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:56.397 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:56.397 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:56.397 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:56.397 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:56.397 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:56.398 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:56.398 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:56.398 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:56.398 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:56.398 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:56.407 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-92: 2019-12-09T23:00:56.407 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:56.407 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:56.407 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:56.407 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:56.408 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:56.408 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:56.408 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:56.408 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:56.408 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:56.408 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:56.409 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:56.409 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:56.409 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:00:56.675 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-94: 2019-12-09T23:00:57.115 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:00:57.115 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:00:57.115 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:00:57.115 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:00:57.115 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:00:57.116 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:00:57.116 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:00:57.116 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:00:57.116 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:00:57.116 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:00:57.116 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:00:57.117 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:00:57.117 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:01:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:ok 2019-12-09T23:01:10.396 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioned_concurrent_object_create_and_remove ... Exception in thread Thread-101: 2019-12-09T23:01:10.397 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:01:10.398 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:01:10.398 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:01:10.398 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:01:10.398 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:01:10.398 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:01:10.399 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:01:10.399 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:01:10.399 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:01:10.399 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:01:10.400 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:01:10.400 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:01:10.400 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:01:11.023 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-106: 2019-12-09T23:01:11.023 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:01:11.023 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:01:11.023 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:01:11.024 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:01:11.024 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:01:11.024 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:01:11.024 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:01:11.024 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:01:11.025 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:01:11.025 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:01:11.025 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:01:11.025 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:01:11.026 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:01:11.037 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-104: 2019-12-09T23:01:11.037 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:01:11.038 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:01:11.038 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:01:11.038 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:01:11.038 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:01:11.039 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:01:11.039 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:01:11.039 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:01:11.039 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:01:11.039 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:01:11.039 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:01:11.040 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:01:11.040 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:01:11.061 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-105: 2019-12-09T23:01:11.061 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:01:11.061 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:01:11.061 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:01:11.062 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:01:11.062 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:01:11.062 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:01:11.062 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:01:11.063 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:01:11.063 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:01:11.063 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:01:11.063 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:01:11.064 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:01:11.064 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:01:11.064 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-102: 2019-12-09T23:01:11.065 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:01:11.065 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:01:11.065 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:01:11.065 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:01:11.066 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:01:11.066 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:01:11.066 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:01:11.066 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:01:11.066 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:01:11.067 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:01:11.067 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:01:11.067 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:01:11.068 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:01:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-107: 2019-12-09T23:01:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:01:11.347 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:01:11.347 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:01:11.347 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:01:11.347 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:01:11.348 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:01:11.348 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:01:11.348 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:01:11.348 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:01:11.348 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:01:11.348 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:01:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:01:11.349 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:01:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-109: 2019-12-09T23:01:11.592 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:01:11.592 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:01:11.593 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:01:11.593 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:01:11.593 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:01:11.593 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:01:11.593 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:01:11.593 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:01:11.593 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:01:11.594 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:01:11.594 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:01:11.594 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:01:11.594 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:01:11.605 INFO:teuthology.orchestra.run.smithi067.stderr:Exception in thread Thread-108: 2019-12-09T23:01:11.605 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last): 2019-12-09T23:01:11.606 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner 2019-12-09T23:01:11.606 INFO:teuthology.orchestra.run.smithi067.stderr: self.run() 2019-12-09T23:01:11.606 INFO:teuthology.orchestra.run.smithi067.stderr: File "/usr/lib64/python2.7/threading.py", line 765, in run 2019-12-09T23:01:11.606 INFO:teuthology.orchestra.run.smithi067.stderr: self.__target(*self.__args, **self.__kwargs) 2019-12-09T23:01:11.606 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests_boto3/functional/test_s3.py", line 8647, in _do_create_object 2019-12-09T23:01:11.607 INFO:teuthology.orchestra.run.smithi067.stderr: client.put_object(Bucket=bucket_name, Key=key, Body=body) 2019-12-09T23:01:11.607 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 272, in _api_call 2019-12-09T23:01:11.607 INFO:teuthology.orchestra.run.smithi067.stderr: return self._make_api_call(operation_name, kwargs) 2019-12-09T23:01:11.607 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.7/site-packages/botocore/client.py", line 576, in _make_api_call 2019-12-09T23:01:11.607 INFO:teuthology.orchestra.run.smithi067.stderr: raise error_class(parsed_response, operation_name) 2019-12-09T23:01:11.607 INFO:teuthology.orchestra.run.smithi067.stderr:ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Unknown 2019-12-09T23:01:11.607 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-09T23:01:23.037 INFO:teuthology.orchestra.run.smithi067.stderr:ok 2019-12-09T23:01:27.001 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set ... ok 2019-12-09T23:01:32.215 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get ... ok 2019-12-09T23:01:36.616 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_get_no_id ... ok 2019-12-09T23:01:38.520 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_id_too_long ... ok 2019-12-09T23:01:41.627 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_same_id ... ok 2019-12-09T23:01:45.144 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_invalid_status ... ok 2019-12-09T23:01:50.400 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_date ... ok 2019-12-09T23:01:52.264 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_invalid_date ... ok 2019-12-09T23:01:56.372 INFO:teuthology.orchestra.run.smithi067.stderr:Check for valid x-amz-expiration header after PUT ... ok 2019-12-09T23:02:02.482 INFO:teuthology.orchestra.run.smithi067.stderr:Check for valid x-amz-expiration header on HEAD request ... ok 2019-12-09T23:02:08.429 INFO:teuthology.orchestra.run.smithi067.stderr:Check for valid x-amz-expiration header on GET request ... ok 2019-12-09T23:02:15.929 INFO:teuthology.orchestra.run.smithi067.stderr:Check x-amz-expiration-header object tags interaction ... ok 2019-12-09T23:02:23.354 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_noncurrent ... ok 2019-12-09T23:02:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_deletemarker ... ok 2019-12-09T23:02:32.068 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_filter ... ok 2019-12-09T23:02:37.017 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_empty_filter ... ok 2019-12-09T23:02:40.882 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_lifecycle_set_multipart ... ok 2019-12-09T23:02:43.538 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1b ... ok 2019-12-09T23:02:48.681 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1kb ... ok 2019-12-09T23:02:53.013 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_1MB ... ok 2019-12-09T23:02:56.345 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encrypted_transfer_13b ... ok 2019-12-09T23:03:01.728 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_method_head ... ok 2019-12-09T23:03:06.358 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_present ... ok 2019-12-09T23:03:10.131 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_other_key ... ok 2019-12-09T23:03:12.547 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_invalid_md5 ... ok 2019-12-09T23:03:15.705 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_md5 ... ok 2019-12-09T23:03:18.191 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_no_key ... ok 2019-12-09T23:03:20.039 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_key_no_sse_c ... ok 2019-12-09T23:04:48.545 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_upload ... ok 2019-12-09T23:04:51.406 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:31:12.472 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:51.413 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:31:12.479 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:51.419 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:31:12.486 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:51.423 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:31:12.489 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:51.435 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:31:12.593 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:51.445 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:31:12.602 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:51.629 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:31:12.694 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:51.629 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:31:12.694 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:51.711 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:31:12.869 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:53.523 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:31:14.589 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:53.526 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:31:14.592 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:53.597 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:31:14.663 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:53.603 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:31:14.669 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:53.605 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:31:14.763 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:53.605 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:31:14.672 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:53.614 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:31:14.772 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:53.620 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:31:14.686 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:04:53.640 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:31:14.798 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:05:29.637 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_multipart_bad_download ... ok 2019-12-09T23:05:33.610 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_encryption_sse_c_post_object_authenticated_request ... ok 2019-12-09T23:05:37.309 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_method_head ... ok 2019-12-09T23:05:41.843 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_present ... ok 2019-12-09T23:05:43.708 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_no_key ... ok 2019-12-09T23:05:45.634 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_not_declared ... ok 2019-12-09T23:06:58.435 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_upload ... ok 2019-12-09T23:07:26.522 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_1 ... ok 2019-12-09T23:08:00.649 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_multipart_invalid_chunks_2 ... ok 2019-12-09T23:08:05.065 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_post_object_authenticated_request ... ok 2019-12-09T23:08:09.024 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1b ... ok 2019-12-09T23:08:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1kb ... ok 2019-12-09T23:08:19.541 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_1MB ... ok 2019-12-09T23:08:22.998 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_transfer_13b ... ok 2019-12-09T23:08:27.261 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_sse_kms_read_declare ... ok 2019-12-09T23:08:34.062 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy ... ok 2019-12-09T23:08:39.114 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy ... ok 2019-12-09T23:08:48.050 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_acl ... ok 2019-12-09T23:08:56.188 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_acl ... ok 2019-12-09T23:09:05.934 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_another_bucket ... ok 2019-12-09T23:09:15.615 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucketv2_policy_another_bucket ... ok 2019-12-09T23:09:22.267 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_obj_tagging ... ok 2019-12-09T23:09:27.671 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_obj_head_tagging ... ok 2019-12-09T23:09:35.191 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_max_tags ... ok 2019-12-09T23:09:40.576 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_excess_tags ... ok 2019-12-09T23:09:46.859 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_max_kvsize_tags ... ok 2019-12-09T23:09:54.564 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_excess_key_tags ... ok 2019-12-09T23:10:01.392 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_excess_val_tags ... ok 2019-12-09T23:10:03.529 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:36:24.594 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:10:03.536 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:36:24.602 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:10:03.544 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:36:24.610 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:10:03.555 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:36:24.614 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:10:03.555 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:36:24.713 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:10:03.576 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:36:24.734 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:10:03.772 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:36:24.929 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:10:03.794 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:36:24.860 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:10:03.797 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:36:24.863 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:10:09.392 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_modify_tags ... ok 2019-12-09T23:10:17.147 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_delete_tags ... ok 2019-12-09T23:10:22.205 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_anonymous_request ... ok 2019-12-09T23:10:24.484 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_post_object_tags_authenticated_request ... ok 2019-12-09T23:10:28.314 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_obj_with_tags ... ok 2019-12-09T23:10:35.942 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_tags_acl_public ... ok 2019-12-09T23:10:43.343 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_tags_acl_public ... ok 2019-12-09T23:10:53.498 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_delete_tags_obj_public ... ok 2019-12-09T23:11:09.944 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_atomic_upload_return_version_id ... ok 2019-12-09T23:12:38.175 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_versioning_bucket_multipart_upload_return_version_id ... ok 2019-12-09T23:12:49.240 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_existing_tag ... ok 2019-12-09T23:13:00.196 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_tagging_existing_tag ... ok 2019-12-09T23:13:09.927 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_tagging_existing_tag ... ok 2019-12-09T23:13:15.100 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:39:36.166 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:13:15.125 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:39:36.190 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:13:15.180 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:39:36.338 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:13:15.182 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:39:36.340 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:13:15.191 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:39:36.257 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:13:15.194 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:39:36.260 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:13:15.253 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:39:36.319 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:13:15.302 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:39:36.367 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:13:15.303 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:39:36.461 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:13:22.413 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source ... ok 2019-12-09T23:13:35.944 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_copy_source_meta ... ok 2019-12-09T23:13:40.010 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_acl ... ok 2019-12-09T23:13:50.365 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_put_obj_grant ... ok 2019-12-09T23:14:01.302 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_bucket_policy_get_obj_acl_existing_tag ... ok 2019-12-09T23:14:07.673 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock ... ok 2019-12-09T23:14:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_bucket ... ok 2019-12-09T23:14:12.025 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_with_days_and_years ... ok 2019-12-09T23:14:14.368 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_days ... ok 2019-12-09T23:14:17.953 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_years ... ok 2019-12-09T23:14:20.879 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_lock_invalid_status ... ok 2019-12-09T23:14:23.549 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_suspend_versioning ... ok 2019-12-09T23:14:26.167 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock ... ok 2019-12-09T23:14:29.339 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_lock_invalid_bucket ... ok 2019-12-09T23:14:42.225 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention ... ok 2019-12-09T23:14:45.800 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_bucket ... ok 2019-12-09T23:14:53.612 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_invalid_mode ... ok 2019-12-09T23:15:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention ... ok 2019-12-09T23:15:09.105 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_retention_invalid_bucket ... ok 2019-12-09T23:15:13.613 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:41:34.678 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:15:13.620 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:41:34.686 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:15:13.628 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:41:34.786 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:15:13.630 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:41:34.695 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:15:13.631 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:41:34.789 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:15:13.636 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:41:34.702 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:15:13.740 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:41:34.805 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:15:13.754 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:41:34.819 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:15:13.762 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:41:34.919 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:15:19.650 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_versionid ... ok 2019-12-09T23:15:31.179 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_override_default_retention ... ok 2019-12-09T23:15:44.806 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_increase_period ... ok 2019-12-09T23:15:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period ... ok 2019-12-09T23:16:01.491 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_obj_retention_shorten_period_bypass ... ok 2019-12-09T23:16:11.467 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_retention ... ok 2019-12-09T23:16:17.642 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold ... ok 2019-12-09T23:16:21.697 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_bucket ... ok 2019-12-09T23:16:27.032 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_put_legal_hold_invalid_status ... ok 2019-12-09T23:16:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold ... ok 2019-12-09T23:16:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_legal_hold_invalid_bucket ... ok 2019-12-09T23:16:46.012 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_on ... ok 2019-12-09T23:16:54.084 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_delete_object_with_legal_hold_off ... ok 2019-12-09T23:17:05.843 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_get_obj_metadata ... ok 2019-12-09T23:17:15.376 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_object_lock_uploading_obj ... ok 2019-12-09T23:17:18.193 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_bucket_policy_status ... ok 2019-12-09T23:17:20.733 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_public_acl_bucket_policy_status ... ok 2019-12-09T23:17:23.971 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_authpublic_acl_bucket_policy_status ... ok 2019-12-09T23:17:27.337 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_publicpolicy_acl_bucket_policy_status ... ok 2019-12-09T23:17:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_acl_bucket_policy_status ... ok 2019-12-09T23:17:36.677 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_nonpublicpolicy_deny_bucket_policy_status ... ok 2019-12-09T23:17:39.375 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_get_default_public_block ... ok 2019-12-09T23:17:43.354 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_put_public_block ... ok 2019-12-09T23:17:46.639 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_block_public_put_bucket_acls ... ok 2019-12-09T23:17:50.997 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_block_public_object_canned_acls ... ok 2019-12-09T23:17:54.035 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_block_public_policy ... ok 2019-12-09T23:18:01.807 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_ignore_public_acls ... ok 2019-12-09T23:18:06.194 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_s3.test_user_policy ... ok 2019-12-09T23:18:06.216 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.functional.test_utils.test_generate ... ok 2019-12-09T23:20:25.873 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:46:46.939 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:20:25.884 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:46:46.950 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:20:25.892 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:46:46.958 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:20:25.900 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:46:46.966 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:20:25.904 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:46:47.061 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:20:25.922 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:46:47.079 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:20:26.084 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:46:47.150 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:20:26.089 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:46:47.155 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:20:26.192 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:46:47.349 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:21:36.806 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:47:57.872 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:21:36.810 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:47:57.876 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:21:36.820 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:47:57.885 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:21:36.854 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:47:58.012 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:21:36.857 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:47:57.923 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:21:36.857 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:47:58.014 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:21:36.862 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:47:57.928 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:21:36.902 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:47:57.968 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:21:36.912 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:47:58.069 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:25:38.656 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:51:59.722 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:25:38.669 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:51:59.728 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:25:38.669 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:51:59.735 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:25:38.673 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:51:59.739 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:25:38.685 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:51:59.841 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:25:38.692 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:51:59.849 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:25:38.838 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:51:59.904 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:25:38.859 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:51:59.925 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:25:38.926 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:52:00.082 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:03.442 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:56:24.508 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:03.516 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:56:24.580 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:03.525 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:56:24.592 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:03.531 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:56:24.597 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:03.537 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:56:24.694 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:03.542 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:56:24.699 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:03.640 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:56:24.707 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:03.653 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:56:24.720 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:03.684 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:56:24.841 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:50.232 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:57:11.299 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:50.239 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:57:11.306 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:50.246 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:57:11.312 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:50.249 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:57:11.406 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:50.251 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:57:11.318 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:50.265 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:57:11.422 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:50.289 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:02:57:11.356 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:50.303 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:02:57:11.370 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:30:50.350 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:02:57:11.507 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:02.355 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:02:23.421 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:02.374 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:02:23.527 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:02.374 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:02:23.428 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:02.375 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:02:23.435 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:02.375 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:02:23.438 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:02.380 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:02:23.537 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:02.564 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:02:23.721 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:02.573 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:02:23.639 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:02.591 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:02:23.657 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:49.932 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:03:10.999 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:49.966 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:03:11.032 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:49.975 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:03:11.041 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:49.993 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:03:11.058 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:50.007 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:03:11.164 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:50.016 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:03:11.170 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:50.028 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:03:11.094 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:50.051 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:03:11.117 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:36:50.057 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:03:11.214 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:38:25.306 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:04:46.372 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:38:25.327 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:04:46.393 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:38:25.328 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:04:46.393 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:38:25.362 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:04:46.428 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:38:25.364 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:04:46.521 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:38:25.384 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:04:46.541 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:38:25.419 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:04:46.485 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:38:25.426 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:04:46.492 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:38:25.456 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:04:46.612 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:41:12.517 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:07:33.579 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:41:12.526 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:07:33.584 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:41:12.527 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:07:33.592 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:41:12.531 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:07:33.597 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:41:12.539 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:07:33.696 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:41:12.552 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:07:33.709 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:41:12.641 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:07:33.707 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:41:12.658 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:07:33.724 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:41:12.693 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:07:33.850 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:25.537 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:12:46.601 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:25.543 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:12:46.609 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:25.554 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:12:46.619 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:25.561 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:12:46.717 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:25.561 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:12:46.625 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:25.563 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:12:46.720 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:25.758 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:12:46.823 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:25.773 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:12:46.928 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:25.822 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:12:46.887 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:46.835 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:13:07.900 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:46.876 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:13:07.941 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:46.906 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:13:07.972 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:46.911 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:13:07.975 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:46.911 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:13:07.976 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:46.917 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:13:08.074 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:46.924 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:13:08.080 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:46.946 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:13:08.011 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:46:46.947 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:13:08.104 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:51:34.947 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:17:56.012 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:51:34.962 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:17:56.020 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:51:34.963 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:17:56.028 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:51:34.967 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:17:56.030 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:51:34.967 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:17:56.125 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:51:34.979 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:17:56.136 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:51:35.171 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:17:56.236 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:51:35.215 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:17:56.372 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:51:35.249 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:17:56.314 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:55:08.386 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:21:29.450 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:55:08.402 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:21:29.456 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:55:08.431 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:21:29.495 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:55:08.432 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:21:29.497 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:55:08.474 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:21:29.631 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:55:08.479 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:21:29.636 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:55:08.523 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:21:29.587 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:55:08.564 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:21:29.626 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:55:08.665 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:21:29.822 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:56:47.118 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:23:08.183 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:56:47.122 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:23:08.186 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:56:47.156 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:23:08.221 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:56:47.169 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:23:08.234 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:56:47.195 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:23:08.352 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:56:47.197 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:23:08.354 8820== Warning: unimplemented fcntl command: 1036 2019-12-09T23:56:47.248 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:23:08.313 9310== Warning: unimplemented fcntl command: 1036 2019-12-09T23:56:47.251 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:23:08.316 9309== Warning: unimplemented fcntl command: 1036 2019-12-09T23:56:47.294 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:23:08.451 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:01:57.310 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:28:18.374 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:01:57.312 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:28:18.377 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:01:57.352 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:28:18.417 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:01:57.356 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:28:18.421 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:01:57.361 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:28:18.518 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:01:57.364 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:28:18.521 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:01:57.514 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:28:18.578 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:01:57.540 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:28:18.605 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:01:57.589 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:28:18.746 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:03:29.918 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:29:50.982 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:03:29.933 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:29:50.997 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:03:29.976 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:29:51.133 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:03:29.978 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:29:51.135 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:03:30.016 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:29:51.080 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:03:30.021 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:29:51.085 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:03:30.023 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:29:51.088 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:03:30.045 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:29:51.202 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:03:30.078 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:29:51.143 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:11:51.660 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:38:12.724 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:11:51.679 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:38:12.744 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:11:51.692 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:38:12.756 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:11:51.693 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:38:12.850 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:11:51.700 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:38:12.765 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:11:51.739 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:38:12.896 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:11:51.932 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:38:12.997 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:11:51.933 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:38:12.997 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:11:51.986 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:38:13.142 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:12:19.729 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:38:40.794 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:12:19.734 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:38:40.799 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:12:19.747 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:38:40.812 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:12:19.752 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:38:40.817 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:12:19.755 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:38:40.912 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:12:19.767 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:38:40.924 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:12:19.787 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:38:40.851 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:12:19.793 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:38:40.857 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:12:19.834 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:38:40.991 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:17:31.814 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:43:52.878 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:17:31.831 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:43:52.885 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:17:31.832 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:43:52.892 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:17:31.833 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:43:52.898 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:17:31.840 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:43:52.997 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:17:31.843 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:43:53.000 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:17:32.035 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:43:53.099 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:17:32.035 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:43:53.100 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:17:32.059 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:43:53.216 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:20:13.586 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:46:34.651 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:20:13.638 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:46:34.703 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:20:13.651 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:46:34.715 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:20:13.653 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:46:34.718 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:20:13.678 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:46:34.834 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:20:13.681 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:46:34.838 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:20:13.755 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:46:34.820 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:20:13.766 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:46:34.831 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:20:13.785 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:46:34.942 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:22:41.979 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:49:03.043 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:22:41.983 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:49:03.047 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:22:41.999 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:49:03.156 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:22:42.003 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:49:03.160 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:22:42.009 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:49:03.073 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:22:42.013 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:49:03.077 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:22:42.107 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:49:03.264 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:22:42.108 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:49:03.172 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:22:42.129 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:49:03.194 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:27:54.988 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:54:16.045 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:27:55.104 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:54:16.164 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:27:55.105 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:54:16.170 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:27:55.105 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:54:16.058 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:27:55.105 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:54:16.063 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:27:55.105 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:54:16.065 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:27:55.198 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:54:16.354 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:27:55.201 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:54:16.265 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:27:55.215 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:54:16.279 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:28:35.082 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:54:56.146 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:28:35.149 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:54:56.213 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:28:35.184 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:54:56.341 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:28:35.188 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:54:56.250 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:28:35.189 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:54:56.345 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:28:35.198 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:54:56.262 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:28:35.218 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:54:56.282 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:28:35.243 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:54:56.400 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:28:35.257 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:54:56.321 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:33:05.363 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:59:26.427 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:33:05.373 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:59:26.437 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:33:05.383 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:59:26.447 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:33:05.385 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:59:26.542 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:33:05.389 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:59:26.453 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:33:05.396 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:59:26.552 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:33:05.577 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:03:59:26.640 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:33:05.590 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:03:59:26.747 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:33:05.616 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:03:59:26.679 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:36:56.830 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:03:17.893 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:36:56.836 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:03:17.900 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:36:56.908 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:03:17.971 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:36:56.909 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:03:17.973 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:36:56.918 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:03:18.074 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:36:56.925 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:03:18.080 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:36:56.948 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:03:18.011 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:36:57.042 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:03:18.102 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:36:57.048 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:03:18.204 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:38:14.535 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:04:35.598 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:38:14.545 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:04:35.609 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:38:14.554 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:04:35.618 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:38:14.561 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:04:35.624 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:38:14.570 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:04:35.726 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:38:14.574 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:04:35.731 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:38:14.636 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:04:35.699 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:38:14.637 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:04:35.701 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:38:14.689 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:04:35.845 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:43:26.677 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:09:47.736 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:43:26.680 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:09:47.744 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:43:26.693 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:09:47.755 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:43:26.694 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:09:47.757 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:43:26.704 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:09:47.860 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:43:26.725 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:09:47.879 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:43:26.883 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:09:47.946 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:43:26.902 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:09:47.965 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:43:26.972 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:09:48.128 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:45:18.304 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:11:39.367 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:45:18.324 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:11:39.387 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:45:18.347 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:11:39.411 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:45:18.357 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:11:39.420 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:45:18.390 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:11:39.547 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:45:18.397 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:11:39.554 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:45:18.415 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:11:39.479 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:45:18.432 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:11:39.483 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:45:18.485 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:11:39.642 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:05.473 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_load_graph ... ok 2019-12-10T00:48:05.477 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_descend_leaf_node ... ok 2019-12-10T00:48:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_descend_node ... ok 2019-12-10T00:48:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_descend_bad_node ... ok 2019-12-10T00:48:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_descend_nonexistant_child ... ok 2019-12-10T00:48:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_printable ... ok 2019-12-10T00:48:05.479 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_binary ... ok 2019-12-10T00:48:05.855 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_printable_no_whitespace ... ok 2019-12-10T00:48:06.180 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_binary_no_whitespace ... ok 2019-12-10T00:48:06.541 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_no_args ... ok 2019-12-10T00:48:06.566 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_no_charset ... ok 2019-12-10T00:48:06.590 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_exact_length ... ok 2019-12-10T00:48:06.590 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_bad_charset ... ok 2019-12-10T00:48:06.590 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_random_missing_length ... ok 2019-12-10T00:48:06.590 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_assemble_decision ... ok 2019-12-10T00:48:06.591 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_escape ... ok 2019-12-10T00:48:06.591 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_indirect ... ok 2019-12-10T00:48:06.591 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_indirect_double ... ok 2019-12-10T00:48:06.591 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_recursive ... ok 2019-12-10T00:48:06.591 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_recursive_mutual ... ok 2019-12-10T00:48:06.591 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_recursive_not_too_eager ... ok 2019-12-10T00:48:06.592 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_make_choice_unweighted_with_space ... ok 2019-12-10T00:48:06.599 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_weighted_choices ... ok 2019-12-10T00:48:06.599 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_null_choices ... ok 2019-12-10T00:48:06.599 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_weighted_null_choices ... ok 2019-12-10T00:48:06.600 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_null_child ... ok 2019-12-10T00:48:06.602 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_weighted_set ... ok 2019-12-10T00:48:06.603 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_header_presence ... ok 2019-12-10T00:48:06.603 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_duplicate_header ... ok 2019-12-10T00:48:06.603 INFO:teuthology.orchestra.run.smithi067.stderr:s3tests_boto3.fuzz.test.test_fuzzer.test_expand_headers ... ok 2019-12-10T00:48:06.625 INFO:teuthology.orchestra.run.smithi067.stderr:test_realistic.TestFileValidator.test_new_file_is_valid ... ok 2019-12-10T00:48:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_on_several_calls ... ok 2019-12-10T00:48:06.647 INFO:teuthology.orchestra.run.smithi067.stderr:test_realistic.TestFileValidator.test_new_file_is_valid_when_size_is_1 ... ok 2019-12-10T00:48:06.805 INFO:teuthology.orchestra.run.smithi067.stderr:test_realistic.TestFiles.test_random_file_valid ... ok 2019-12-10T00:48:06.806 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-10T00:48:06.806 INFO:teuthology.orchestra.run.smithi067.stderr:---------------------------------------------------------------------- 2019-12-10T00:48:06.806 INFO:teuthology.orchestra.run.smithi067.stderr:Ran 627 tests in 11576.630s 2019-12-10T00:48:06.806 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-10T00:48:06.807 INFO:teuthology.orchestra.run.smithi067.stderr:OK (SKIP=75) 2019-12-10T00:48:06.857 DEBUG:tasks.s3tests:Scanning radosgw logs for leaked encryption keys... 2019-12-10T00:48:06.858 INFO:teuthology.orchestra.run.smithi067:> grep --binary-files=text pO3upElrwuEXSoFwCfnZPdSsmt/xWeFa0N9KgDijwVs= /var/log/ceph/rgw.ceph.client.0.log 2019-12-10T00:48:06.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-10T00:48:06.960 INFO:tasks.s3tests:Cleaning up boto... 2019-12-10T00:48:06.961 INFO:teuthology.orchestra.run.smithi067:> rm /home/ubuntu/cephtest/boto.cfg 2019-12-10T00:48:07.004 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data --cluster ceph 2019-12-10T00:48:07.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.175+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.200+0000 7fa59001dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.200+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.209+0000 7fa59001dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.209+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.213+0000 7fa59001dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.213+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.222 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.222+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.392+0000 7fa59001dec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-10T00:48:07.392 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.392+0000 7fa59001dec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-10T00:48:07.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.392+0000 7fa59001dec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-10T00:48:07.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.392+0000 7fa59001dec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-10T00:48:07.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.392+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.402+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:07.404 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.402+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.410+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:07.410 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.410+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.418+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:07.421 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.418+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.427 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.427+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:07.427 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.427+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.434 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.434+0000 7fa59001dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.434+0000 7fa59001dec0 10 cannot find current period zonegroup using local zonegroup 2019-12-10T00:48:07.435 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.434+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.439+0000 7fa59001dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:07.439 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.439+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.447 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.447+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:07.447 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.447+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.458+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.458+0000 7fa59001dec0 20 zonegroup default 2019-12-10T00:48:07.458 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.458+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.464+0000 7fa59001dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.464+0000 7fa59001dec0 10 Cannot find current period zone using local zone 2019-12-10T00:48:07.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.464+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.469+0000 7fa59001dec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:07.469 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.469+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.476 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.476+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:07.476 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.476+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:07.484 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.484+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:07.485 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.484+0000 7fa59001dec0 20 zone default found 2019-12-10T00:48:07.485 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.484+0000 7fa59001dec0 20 started sync module instance, tier type = 2019-12-10T00:48:07.485 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:07.484+0000 7fa59001dec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-10T00:48:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.014+0000 7fa59001dec0 20 add_watcher() i=0 2019-12-10T00:48:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.014+0000 7fa59001dec0 20 add_watcher() i=1 2019-12-10T00:48:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.014+0000 7fa59001dec0 20 add_watcher() i=2 2019-12-10T00:48:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.014+0000 7fa59001dec0 20 add_watcher() i=3 2019-12-10T00:48:08.016 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.014+0000 7fa59001dec0 20 add_watcher() i=4 2019-12-10T00:48:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.014+0000 7fa59001dec0 20 add_watcher() i=5 2019-12-10T00:48:08.017 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.014+0000 7fa59001dec0 20 add_watcher() i=6 2019-12-10T00:48:08.051 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.051+0000 7fa59001dec0 20 add_watcher() i=7 2019-12-10T00:48:08.051 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.051+0000 7fa59001dec0 2 all 8 watchers are set, enabling cache 2019-12-10T00:48:08.051 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.051+0000 7fa54dfdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T00:48:08.064 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.064+0000 7fa57d247700 0 -- 172.21.15.67:0/2567900269 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557be4984cd0 msgr2=0x557be498b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T00:48:08.092 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:08.092+0000 7fa577fff700 0 -- 172.21.15.67:0/1264493267 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557be4991e50 msgr2=0x557be4994320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T00:48:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.747+0000 7fa4bffff700 20 reqs_thread_entry: start 2019-12-10T00:48:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.747+0000 7fa4bdffb700 20 reqs_thread_entry: start 2019-12-10T00:48:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.748+0000 7fa59001dec0 20 init_complete bucket index max shards: 0 2019-12-10T00:48:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.748+0000 7fa4abfff700 20 reqs_thread_entry: start 2019-12-10T00:48:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.748+0000 7fa59001dec0 20 get_system_obj_state: rctx=0x7ffdafc06ae8 obj=default.rgw.meta:users.uid:bar.client.0 state=0x557be4a06d50 s->prefetch_data=0 2019-12-10T00:48:09.749 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.748+0000 7fa59001dec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2019-12-10T00:48:09.767 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.767+0000 7fa59001dec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x16 2019-12-10T00:48:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.767+0000 7fa59001dec0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-10T00:48:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.767+0000 7fa59001dec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-10T00:48:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.767+0000 7fa59001dec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-10T00:48:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.767+0000 7fa59001dec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-10T00:48:09.768 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.767+0000 7fa59001dec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.795+0000 7fa59001dec0 20 rados_obj.operate() r=0 bl.length=465 2019-12-10T00:48:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.795+0000 7fa59001dec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x11 2019-12-10T00:48:09.795 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.795+0000 7fa59001dec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-10T00:48:09.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.810+0000 7fa59001dec0 10 removing key index: ODWNEOWYVZRFIQZWQXXL 2019-12-10T00:48:09.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.810+0000 7fa59001dec0 10 distributing notification oid=default.rgw.control:notify.3 bl.length()=170 2019-12-10T00:48:09.830 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:09.830+0000 7fa55e7fc700 10 RGWWatcher::handle_notify() notify_id 605590388784 cookie 93990604438304 notifier 5058 bl.length()=170 2019-12-10T00:48:10.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:10.243+0000 7fa59001dec0 10 removing email index: bar.client.0+test@test.test 2019-12-10T00:48:10.244 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:10.243+0000 7fa59001dec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=178 2019-12-10T00:48:10.260 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:10.260+0000 7fa55e7fc700 10 RGWWatcher::handle_notify() notify_id 605590389669 cookie 93990604432448 notifier 5058 bl.length()=178 2019-12-10T00:48:10.776 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:10.775+0000 7fa59001dec0 10 removing user buckets index 2019-12-10T00:48:10.776 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:10.775+0000 7fa59001dec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=169 2019-12-10T00:48:10.793 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:10.793+0000 7fa55e7fc700 10 RGWWatcher::handle_notify() notify_id 605590388837 cookie 93990604436240 notifier 5058 bl.length()=169 2019-12-10T00:48:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.118+0000 7fa59001dec0 10 removing user index: bar.client.0 2019-12-10T00:48:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.118+0000 7fa59001dec0 10 removing default.rgw.meta+users.uid+bar.client.0 from cache 2019-12-10T00:48:11.121 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.118+0000 7fa59001dec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=161 2019-12-10T00:48:11.136 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.135+0000 7fa55e7fc700 10 RGWWatcher::handle_notify() notify_id 605590389670 cookie 93990604432448 notifier 5058 bl.length()=161 2019-12-10T00:48:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.317+0000 7fa59001dec0 20 remove_watcher() i=0 2019-12-10T00:48:11.318 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.317+0000 7fa59001dec0 2 removed watcher, disabling cache 2019-12-10T00:48:11.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.364+0000 7fa59001dec0 20 remove_watcher() i=1 2019-12-10T00:48:11.397 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.396+0000 7fa59001dec0 20 remove_watcher() i=2 2019-12-10T00:48:11.471 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.471+0000 7fa59001dec0 20 remove_watcher() i=3 2019-12-10T00:48:11.544 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.543+0000 7fa59001dec0 20 remove_watcher() i=4 2019-12-10T00:48:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.613+0000 7fa59001dec0 20 remove_watcher() i=5 2019-12-10T00:48:11.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.675+0000 7fa59001dec0 20 remove_watcher() i=6 2019-12-10T00:48:11.754 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.754+0000 7fa59001dec0 20 remove_watcher() i=7 2019-12-10T00:48:11.767 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph 2019-12-10T00:48:11.937 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.936+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:11.977 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.973+0000 7f0683b55ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:11.977 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.973+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:11.984 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.983+0000 7f0683b55ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:11.984 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.983+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:11.988 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.988+0000 7f0683b55ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:11.988 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.988+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:11.996+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:12.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.171+0000 7f0683b55ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-10T00:48:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.171+0000 7f0683b55ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-10T00:48:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.171+0000 7f0683b55ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-10T00:48:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.171+0000 7f0683b55ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-10T00:48:12.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.171+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.180 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.180+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:12.181 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.180+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.190+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:12.191 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.191+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.200+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:12.200 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.200+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.210 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.210+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:12.211 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.210+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.215 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.214+0000 7f0683b55ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:12.215 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.214+0000 7f0683b55ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-10T00:48:12.215 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.214+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.221 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.221+0000 7f0683b55ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:12.222 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.221+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.230 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.230+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:12.231 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.230+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.240+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:12.241 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.240+0000 7f0683b55ec0 20 zonegroup default 2019-12-10T00:48:12.241 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.240+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.245+0000 7f0683b55ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:12.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.245+0000 7f0683b55ec0 10 Cannot find current period zone using local zone 2019-12-10T00:48:12.246 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.245+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.253 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.253+0000 7f0683b55ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:12.253 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.253+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.264 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.264+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:12.265 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.264+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:12.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.273+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:12.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.273+0000 7f0683b55ec0 20 zone default found 2019-12-10T00:48:12.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.273+0000 7f0683b55ec0 20 started sync module instance, tier type = 2019-12-10T00:48:12.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.273+0000 7f0683b55ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-10T00:48:12.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.739+0000 7f0683b55ec0 20 add_watcher() i=0 2019-12-10T00:48:12.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.739+0000 7f0683b55ec0 20 add_watcher() i=1 2019-12-10T00:48:12.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.739+0000 7f0683b55ec0 20 add_watcher() i=2 2019-12-10T00:48:12.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.739+0000 7f0683b55ec0 20 add_watcher() i=3 2019-12-10T00:48:12.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.739+0000 7f0683b55ec0 20 add_watcher() i=4 2019-12-10T00:48:12.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.739+0000 7f0683b55ec0 20 add_watcher() i=5 2019-12-10T00:48:12.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.739+0000 7f0683b55ec0 20 add_watcher() i=6 2019-12-10T00:48:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.773+0000 7f0670d7f700 0 -- 172.21.15.67:0/691697087 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x557a58bb98f0 msgr2=0x557a58bbbd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T00:48:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.774+0000 7f0683b55ec0 20 add_watcher() i=7 2019-12-10T00:48:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.774+0000 7f0683b55ec0 2 all 8 watchers are set, enabling cache 2019-12-10T00:48:12.775 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.774+0000 7f06417e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T00:48:12.810 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:12.806+0000 7f0670d7f700 0 -- 172.21.15.67:0/88637823 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x557a58bd0560 msgr2=0x557a58bd2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T00:48:14.276 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.274+0000 7f05b37fe700 20 reqs_thread_entry: start 2019-12-10T00:48:14.276 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.274+0000 7f05b17fa700 20 reqs_thread_entry: start 2019-12-10T00:48:14.276 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.275+0000 7f0683b55ec0 20 init_complete bucket index max shards: 0 2019-12-10T00:48:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.275+0000 7f059f7fe700 20 reqs_thread_entry: start 2019-12-10T00:48:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.275+0000 7f0683b55ec0 20 get_system_obj_state: rctx=0x7ffc105d2bf8 obj=default.rgw.meta:users.uid:foo.client.0 state=0x557a58c442a0 s->prefetch_data=0 2019-12-10T00:48:14.277 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.275+0000 7f0683b55ec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2019-12-10T00:48:14.291 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.289+0000 7f0683b55ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x16 2019-12-10T00:48:14.291 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.289+0000 7f0683b55ec0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-10T00:48:14.291 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.289+0000 7f0683b55ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-10T00:48:14.291 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.289+0000 7f0683b55ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-10T00:48:14.291 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.289+0000 7f0683b55ec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-10T00:48:14.292 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.289+0000 7f0683b55ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.319+0000 7f0683b55ec0 20 rados_obj.operate() r=0 bl.length=465 2019-12-10T00:48:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.319+0000 7f0683b55ec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x11 2019-12-10T00:48:14.320 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.319+0000 7f0683b55ec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-10T00:48:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.338+0000 7f0683b55ec0 10 removing key index: VNARFYYYGGEZXSJHFFIQ 2019-12-10T00:48:14.340 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.338+0000 7f0683b55ec0 10 distributing notification oid=default.rgw.control:notify.2 bl.length()=170 2019-12-10T00:48:14.355 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:14.353+0000 7f06567fc700 10 RGWWatcher::handle_notify() notify_id 605590388838 cookie 93983963029472 notifier 4936 bl.length()=170 2019-12-10T00:48:15.007 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.006+0000 7f0683b55ec0 10 removing email index: foo.client.0+test@test.test 2019-12-10T00:48:15.007 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.006+0000 7f0683b55ec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=178 2019-12-10T00:48:15.026 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.023+0000 7f06567fc700 10 RGWWatcher::handle_notify() notify_id 605590388883 cookie 93983963056592 notifier 4936 bl.length()=178 2019-12-10T00:48:15.153 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.152+0000 7f0683b55ec0 10 removing user buckets index 2019-12-10T00:48:15.154 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.152+0000 7f0683b55ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=169 2019-12-10T00:48:15.165 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.163+0000 7f06567fc700 10 RGWWatcher::handle_notify() notify_id 605590389671 cookie 93983963031888 notifier 4936 bl.length()=169 2019-12-10T00:48:15.441 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.440+0000 7f0683b55ec0 10 removing user index: foo.client.0 2019-12-10T00:48:15.442 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.440+0000 7f0683b55ec0 10 removing default.rgw.meta+users.uid+foo.client.0 from cache 2019-12-10T00:48:15.442 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.440+0000 7f0683b55ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=161 2019-12-10T00:48:15.459 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:15.458+0000 7f06567fc700 10 RGWWatcher::handle_notify() notify_id 605590388864 cookie 93983963028544 notifier 4936 bl.length()=161 2019-12-10T00:48:16.157 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.153+0000 7f0683b55ec0 20 remove_watcher() i=0 2019-12-10T00:48:16.157 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.153+0000 7f0683b55ec0 2 removed watcher, disabling cache 2019-12-10T00:48:16.222 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.221+0000 7f0683b55ec0 20 remove_watcher() i=1 2019-12-10T00:48:16.275 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.274+0000 7f0683b55ec0 20 remove_watcher() i=2 2019-12-10T00:48:16.350 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.349+0000 7f0683b55ec0 20 remove_watcher() i=3 2019-12-10T00:48:16.416 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.415+0000 7f0683b55ec0 20 remove_watcher() i=4 2019-12-10T00:48:16.497 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.496+0000 7f0683b55ec0 20 remove_watcher() i=5 2019-12-10T00:48:16.544 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.543+0000 7f0683b55ec0 20 remove_watcher() i=6 2019-12-10T00:48:16.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.595+0000 7f0683b55ec0 20 remove_watcher() i=7 2019-12-10T00:48:16.604 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid 'testx$tenanteduser.client.0' --purge-data --cluster ceph 2019-12-10T00:48:16.796 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.795+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:16.834 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.833+0000 7f279dcf0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:16.834 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.833+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.850+0000 7f279dcf0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:16.851 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.850+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:16.856 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.855+0000 7f279dcf0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:16.856 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.855+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:16.871 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:16.864+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:17.068 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.067+0000 7f279dcf0ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-10T00:48:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.067+0000 7f279dcf0ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-10T00:48:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.067+0000 7f279dcf0ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-10T00:48:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.067+0000 7f279dcf0ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-10T00:48:17.069 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.067+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.081+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:17.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.081+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.096+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:17.101 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.097+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.109+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:17.110 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.109+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.125 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.121+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:17.125 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.121+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.128+0000 7f279dcf0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.128+0000 7f279dcf0ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-10T00:48:17.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.128+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.137+0000 7f279dcf0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:17.138 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.137+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.150+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:17.153 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.150+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.163 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.162+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.163+0000 7f279dcf0ec0 20 zonegroup default 2019-12-10T00:48:17.164 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.163+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.169+0000 7f279dcf0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:17.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.169+0000 7f279dcf0ec0 10 Cannot find current period zone using local zone 2019-12-10T00:48:17.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.169+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.175+0000 7f279dcf0ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:17.176 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.175+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.187 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.186+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:17.188 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.186+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.202+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.202+0000 7f279dcf0ec0 20 zone default found 2019-12-10T00:48:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.202+0000 7f279dcf0ec0 20 started sync module instance, tier type = 2019-12-10T00:48:17.203 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.202+0000 7f279dcf0ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-10T00:48:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.739+0000 7f279dcf0ec0 20 add_watcher() i=0 2019-12-10T00:48:17.740 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.739+0000 7f279dcf0ec0 20 add_watcher() i=1 2019-12-10T00:48:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.739+0000 7f279dcf0ec0 20 add_watcher() i=2 2019-12-10T00:48:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.739+0000 7f279dcf0ec0 20 add_watcher() i=3 2019-12-10T00:48:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.739+0000 7f279dcf0ec0 20 add_watcher() i=4 2019-12-10T00:48:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.739+0000 7f279dcf0ec0 20 add_watcher() i=5 2019-12-10T00:48:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.739+0000 7f279dcf0ec0 20 add_watcher() i=6 2019-12-10T00:48:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.779+0000 7f279dcf0ec0 20 add_watcher() i=7 2019-12-10T00:48:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.779+0000 7f279dcf0ec0 2 all 8 watchers are set, enabling cache 2019-12-10T00:48:17.780 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.779+0000 7f275b7de700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T00:48:17.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.784+0000 7f278af1a700 0 -- 172.21.15.67:0/2296088392 >> v2:172.21.15.67:3300/0 conn(0x555f64f791b0 msgr2=0x555f64f7b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-10T00:48:17.987 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:17.986+0000 7f278af1a700 0 --2- 172.21.15.67:0/2296088392 >> v2:172.21.15.67:3300/0 conn(0x555f64f791b0 0x555f64f7b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-10T00:48:18.012 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:18.011+0000 7f278b71b700 0 -- 172.21.15.67:0/1914172272 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x555f64f7c1a0 msgr2=0x555f64f7c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T00:48:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.477+0000 7f26c9ffb700 20 reqs_thread_entry: start 2019-12-10T00:48:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.478+0000 7f26c37fe700 20 reqs_thread_entry: start 2019-12-10T00:48:19.479 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.478+0000 7f279dcf0ec0 20 init_complete bucket index max shards: 0 2019-12-10T00:48:19.480 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.478+0000 7f26c1ffb700 20 reqs_thread_entry: start 2019-12-10T00:48:19.480 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.479+0000 7f279dcf0ec0 20 get_system_obj_state: rctx=0x7fffe14cac98 obj=default.rgw.meta:users.uid:testx$tenanteduser.client.0 state=0x555f64ffadd0 s->prefetch_data=0 2019-12-10T00:48:19.480 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.479+0000 7f279dcf0ec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : miss 2019-12-10T00:48:19.492 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.490+0000 7f279dcf0ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x16 2019-12-10T00:48:19.493 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.490+0000 7f279dcf0ec0 10 adding default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-10T00:48:19.493 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.490+0000 7f279dcf0ec0 10 updating xattr: name=ceph.objclass.version bl.length()=42 2019-12-10T00:48:19.493 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.490+0000 7f279dcf0ec0 10 updating xattr: name=user.rgw.user-policy bl.length()=118 2019-12-10T00:48:19.493 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.490+0000 7f279dcf0ec0 20 get_system_obj_state: s->obj_tag was set empty 2019-12-10T00:48:19.493 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.490+0000 7f279dcf0ec0 10 cache get: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 : type miss (requested=0x11, cached=0x16) 2019-12-10T00:48:19.493 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.490+0000 7f279dcf0ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:19.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.504+0000 7f279dcf0ec0 20 rados_obj.operate() r=0 bl.length=524 2019-12-10T00:48:19.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.504+0000 7f279dcf0ec0 10 cache put: name=default.rgw.meta+users.uid+testx$tenanteduser.client.0 info.flags=0x11 2019-12-10T00:48:19.505 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.504+0000 7f279dcf0ec0 10 moving default.rgw.meta+users.uid+testx$tenanteduser.client.0 to cache LRU end 2019-12-10T00:48:19.516 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.515+0000 7f279dcf0ec0 10 removing key index: OCRIEMLAHEQGTBVTWNEP 2019-12-10T00:48:19.516 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.515+0000 7f279dcf0ec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=170 2019-12-10T00:48:19.534 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:19.533+0000 7f276bfff700 10 RGWWatcher::handle_notify() notify_id 605590388865 cookie 93868204093600 notifier 5085 bl.length()=170 2019-12-10T00:48:20.030 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.029+0000 7f279dcf0ec0 10 removing email index: testx$tenanteduser.client.0+test@test.test 2019-12-10T00:48:20.031 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.029+0000 7f279dcf0ec0 10 distributing notification oid=default.rgw.control:notify.7 bl.length()=193 2019-12-10T00:48:20.060 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.058+0000 7f276bfff700 10 RGWWatcher::handle_notify() notify_id 605590388884 cookie 93868204122816 notifier 5085 bl.length()=193 2019-12-10T00:48:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.141+0000 7f279dcf0ec0 10 removing user buckets index 2019-12-10T00:48:20.142 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.141+0000 7f279dcf0ec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=184 2019-12-10T00:48:20.173 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.168+0000 7f276bfff700 10 RGWWatcher::handle_notify() notify_id 605590388885 cookie 93868204121440 notifier 5085 bl.length()=184 2019-12-10T00:48:20.680 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.679+0000 7f279dcf0ec0 10 removing user index: testx$tenanteduser.client.0 2019-12-10T00:48:20.680 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.679+0000 7f279dcf0ec0 10 removing default.rgw.meta+users.uid+testx$tenanteduser.client.0 from cache 2019-12-10T00:48:20.680 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.679+0000 7f279dcf0ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=176 2019-12-10T00:48:20.690 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:20.689+0000 7f276bfff700 10 RGWWatcher::handle_notify() notify_id 605590388839 cookie 93868204118128 notifier 5085 bl.length()=176 2019-12-10T00:48:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:21.171+0000 7f279dcf0ec0 20 remove_watcher() i=0 2019-12-10T00:48:21.173 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:21.171+0000 7f279dcf0ec0 2 removed watcher, disabling cache 2019-12-10T00:48:21.224 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:21.223+0000 7f279dcf0ec0 20 remove_watcher() i=1 2019-12-10T00:48:21.270 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:21.269+0000 7f279dcf0ec0 20 remove_watcher() i=2 2019-12-10T00:48:21.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:21.318+0000 7f279dcf0ec0 20 remove_watcher() i=3 2019-12-10T00:48:21.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:21.365+0000 7f279dcf0ec0 20 remove_watcher() i=4 2019-12-10T00:48:21.428 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:21.426+0000 7f279dcf0ec0 20 remove_watcher() i=5 2019-12-10T00:48:21.479 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:21.478+0000 7f279dcf0ec0 20 remove_watcher() i=6 2019-12-10T00:48:21.554 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:21.553+0000 7f279dcf0ec0 20 remove_watcher() i=7 2019-12-10T00:48:21.562 INFO:tasks.s3tests:Removing s3-tests... 2019-12-10T00:48:21.562 INFO:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/s3-tests 2019-12-10T00:48:21.710 INFO:teuthology.run_tasks:Running task swift... 2019-12-10T00:48:21.729 INFO:tasks.swift:clients=['client.0'] 2019-12-10T00:48:21.729 INFO:tasks.swift:Downloading swift... 2019-12-10T00:48:21.730 INFO:teuthology.orchestra.run.smithi067:> git clone -b ceph-master git://git.ceph.com/git/swift.git /home/ubuntu/cephtest/swift 2019-12-10T00:48:21.838 INFO:teuthology.orchestra.run.smithi067.stdout:Cloning into '/home/ubuntu/cephtest/swift'... 2019-12-10T00:48:22.287 INFO:tasks.swift:Creating rgw users... 2019-12-10T00:48:22.287 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph user create --subuser foo.client.0:foo --display-name 'Mr. foo.client.0 foo' --secret B1wiwLWVdxJepwNPwGRVqkeo0Y/oTI96CQ6b9KA26kn0FzJfuaeSdw== --email foo.client.0+test@test.test --key-type swift --access full 2019-12-10T00:48:22.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.431+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.456 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.455+0000 7fdcd7bccec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:22.456 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.455+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.464 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.463+0000 7fdcd7bccec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:22.465 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.463+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.468 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.467+0000 7fdcd7bccec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:22.469 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.467+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.477 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.476+0000 7fdcd7bccec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:22.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.723+0000 7fdcd7bccec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-10T00:48:22.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.723+0000 7fdcd7bccec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-10T00:48:22.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.723+0000 7fdcd7bccec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-10T00:48:22.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.723+0000 7fdcd7bccec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-10T00:48:22.725 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.723+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.733 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.732+0000 7fdcd7bccec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:22.733 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.732+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.745 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.744+0000 7fdcd7bccec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:22.745 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.744+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.754 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.753+0000 7fdcd7bccec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:22.754 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.753+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.765+0000 7fdcd7bccec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:22.767 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.766+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.771+0000 7fdcd7bccec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.771+0000 7fdcd7bccec0 10 cannot find current period zonegroup using local zonegroup 2019-12-10T00:48:22.772 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.771+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.777 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.776+0000 7fdcd7bccec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:22.777 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.776+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.784+0000 7fdcd7bccec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:22.785 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.784+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.793 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.792+0000 7fdcd7bccec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.792+0000 7fdcd7bccec0 20 zonegroup default 2019-12-10T00:48:22.794 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.792+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.800+0000 7fdcd7bccec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:22.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.800+0000 7fdcd7bccec0 10 Cannot find current period zone using local zone 2019-12-10T00:48:22.801 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.800+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.805+0000 7fdcd7bccec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:22.806 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.805+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.814 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.813+0000 7fdcd7bccec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:22.815 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.813+0000 7fdcd7bccec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.820+0000 7fdcd7bccec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.820+0000 7fdcd7bccec0 20 zone default found 2019-12-10T00:48:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.820+0000 7fdcd7bccec0 20 started sync module instance, tier type = 2019-12-10T00:48:22.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:22.820+0000 7fdcd7bccec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-10T00:48:23.238 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.237+0000 7fdcd7bccec0 20 add_watcher() i=0 2019-12-10T00:48:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.237+0000 7fdcd7bccec0 20 add_watcher() i=1 2019-12-10T00:48:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.237+0000 7fdcd7bccec0 20 add_watcher() i=2 2019-12-10T00:48:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.237+0000 7fdcd7bccec0 20 add_watcher() i=3 2019-12-10T00:48:23.239 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.237+0000 7fdcd7bccec0 20 add_watcher() i=4 2019-12-10T00:48:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.237+0000 7fdcd7bccec0 20 add_watcher() i=5 2019-12-10T00:48:23.240 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.237+0000 7fdcd7bccec0 20 add_watcher() i=6 2019-12-10T00:48:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.273+0000 7fdcd7bccec0 20 add_watcher() i=7 2019-12-10T00:48:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.273+0000 7fdcd7bccec0 2 all 8 watchers are set, enabling cache 2019-12-10T00:48:23.274 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.273+0000 7fdc957e2700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T00:48:23.277 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.276+0000 7fdcc4df6700 0 -- 172.21.15.67:0/1961895523 >> v1:172.21.15.81:6789/0 conn(0x5555ae4eee40 legacy=0x5555ae4f1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until injecting socket failure 2019-12-10T00:48:23.312 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:23.311+0000 7fdcc4df6700 0 -- 172.21.15.67:0/323501756 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x5555ae503ea0 msgr2=0x5555ae5062c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T00:48:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.363+0000 7fdc077fe700 20 reqs_thread_entry: start 2019-12-10T00:48:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.364+0000 7fdc057fa700 20 reqs_thread_entry: start 2019-12-10T00:48:25.365 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.364+0000 7fdcd7bccec0 20 init_complete bucket index max shards: 0 2019-12-10T00:48:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.364+0000 7fdbf37fe700 20 reqs_thread_entry: start 2019-12-10T00:48:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.364+0000 7fdcd7bccec0 20 get_system_obj_state: rctx=0x7ffde3cbd928 obj=default.rgw.meta:users.uid:foo.client.0 state=0x5555ae570910 s->prefetch_data=0 2019-12-10T00:48:25.366 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.364+0000 7fdcd7bccec0 10 cache get: name=default.rgw.meta+users.uid+foo.client.0 : miss 2019-12-10T00:48:25.378 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.376+0000 7fdcd7bccec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x0 2019-12-10T00:48:25.378 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.376+0000 7fdcd7bccec0 10 adding default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-10T00:48:25.378 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.376+0000 7fdcd7bccec0 20 get_system_obj_state: rctx=0x7ffde3cbd938 obj=default.rgw.meta:users.email:foo.client.0+test@test.test state=0x5555ae5726a0 s->prefetch_data=0 2019-12-10T00:48:25.378 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.376+0000 7fdcd7bccec0 10 cache get: name=default.rgw.meta+users.email+foo.client.0+test@test.test : miss 2019-12-10T00:48:25.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.383+0000 7fdcd7bccec0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x0 2019-12-10T00:48:25.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.383+0000 7fdcd7bccec0 10 adding default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-10T00:48:25.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.383+0000 7fdcd7bccec0 20 get_system_obj_state: rctx=0x7ffde3cbd548 obj=default.rgw.meta:users.swift:foo.client.0:foo state=0x5555ae5726a0 s->prefetch_data=0 2019-12-10T00:48:25.384 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.383+0000 7fdcd7bccec0 10 cache get: name=default.rgw.meta+users.swift+foo.client.0:foo : miss 2019-12-10T00:48:25.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.390+0000 7fdcd7bccec0 10 cache put: name=default.rgw.meta+users.swift+foo.client.0:foo info.flags=0x0 2019-12-10T00:48:25.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.390+0000 7fdcd7bccec0 10 adding default.rgw.meta+users.swift+foo.client.0:foo to cache LRU end 2019-12-10T00:48:25.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.390+0000 7fdcd7bccec0 20 get_system_obj_state: rctx=0x7ffde3cbd6c8 obj=default.rgw.meta:users.swift:foo.client.0:foo state=0x5555ae572950 s->prefetch_data=0 2019-12-10T00:48:25.391 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.390+0000 7fdcd7bccec0 10 cache get: name=default.rgw.meta+users.swift+foo.client.0:foo : type miss (requested=0x6, cached=0x0) 2019-12-10T00:48:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.397+0000 7fdcd7bccec0 10 cache put: name=default.rgw.meta+users.swift+foo.client.0:foo info.flags=0x0 2019-12-10T00:48:25.398 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.397+0000 7fdcd7bccec0 10 moving default.rgw.meta+users.swift+foo.client.0:foo to cache LRU end 2019-12-10T00:48:25.477 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.476+0000 7fdcd7bccec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-10T00:48:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.476+0000 7fdcd7bccec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-10T00:48:25.478 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.476+0000 7fdcd7bccec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=612 2019-12-10T00:48:25.492 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.491+0000 7fdcaa7fc700 10 RGWWatcher::handle_notify() notify_id 605590388866 cookie 93826484882736 notifier 4661 bl.length()=612 2019-12-10T00:48:25.492 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.491+0000 7fdcaa7fc700 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-10T00:48:25.492 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.491+0000 7fdcaa7fc700 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-10T00:48:25.678 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.676+0000 7fdcd7bccec0 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2019-12-10T00:48:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.676+0000 7fdcd7bccec0 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-10T00:48:25.679 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.676+0000 7fdcd7bccec0 10 distributing notification oid=default.rgw.control:notify.6 bl.length()=194 2019-12-10T00:48:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.698+0000 7fdcaa7fc700 10 RGWWatcher::handle_notify() notify_id 605590388886 cookie 93826484924192 notifier 4661 bl.length()=194 2019-12-10T00:48:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.698+0000 7fdcaa7fc700 10 cache put: name=default.rgw.meta+users.email+foo.client.0+test@test.test info.flags=0x7 2019-12-10T00:48:25.699 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.698+0000 7fdcaa7fc700 10 moving default.rgw.meta+users.email+foo.client.0+test@test.test to cache LRU end 2019-12-10T00:48:25.861 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.860+0000 7fdcd7bccec0 10 cache put: name=default.rgw.meta+users.swift+foo.client.0:foo info.flags=0x7 2019-12-10T00:48:25.861 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.860+0000 7fdcd7bccec0 10 moving default.rgw.meta+users.swift+foo.client.0:foo to cache LRU end 2019-12-10T00:48:25.861 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.860+0000 7fdcd7bccec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=183 2019-12-10T00:48:25.874 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.873+0000 7fdcaa7fc700 10 RGWWatcher::handle_notify() notify_id 605590389672 cookie 93826484886176 notifier 4661 bl.length()=183 2019-12-10T00:48:25.875 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.873+0000 7fdcaa7fc700 10 cache put: name=default.rgw.meta+users.swift+foo.client.0:foo info.flags=0x7 2019-12-10T00:48:25.875 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:25.873+0000 7fdcaa7fc700 10 moving default.rgw.meta+users.swift+foo.client.0:foo to cache LRU end 2019-12-10T00:48:26.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.323+0000 7fdcd7bccec0 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-10T00:48:26.324 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.323+0000 7fdcd7bccec0 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-10T00:48:26.325 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.323+0000 7fdcd7bccec0 10 distributing notification oid=default.rgw.control:notify.0 bl.length()=636 2019-12-10T00:48:26.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.343+0000 7fdcaa7fc700 10 RGWWatcher::handle_notify() notify_id 605590388867 cookie 93826484882736 notifier 4661 bl.length()=636 2019-12-10T00:48:26.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.343+0000 7fdcaa7fc700 10 cache put: name=default.rgw.meta+users.uid+foo.client.0 info.flags=0x17 2019-12-10T00:48:26.344 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.343+0000 7fdcaa7fc700 10 moving default.rgw.meta+users.uid+foo.client.0 to cache LRU end 2019-12-10T00:48:26.658 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2019-12-10T00:48:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "user_id": "foo.client.0", 2019-12-10T00:48:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "display_name": "Mr. foo.client.0 foo", 2019-12-10T00:48:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "email": "foo.client.0+test@test.test", 2019-12-10T00:48:26.658 INFO:teuthology.orchestra.run.smithi067.stdout: "suspended": 0, 2019-12-10T00:48:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "max_buckets": 1000, 2019-12-10T00:48:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "subusers": [ 2019-12-10T00:48:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-10T00:48:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "id": "foo.client.0:foo", 2019-12-10T00:48:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: "permissions": "full-control" 2019-12-10T00:48:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-10T00:48:26.659 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-10T00:48:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "keys": [], 2019-12-10T00:48:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "swift_keys": [ 2019-12-10T00:48:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-10T00:48:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "user": "foo.client.0:foo", 2019-12-10T00:48:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: "secret_key": "B1wiwLWVdxJepwNPwGRVqkeo0Y/oTI96CQ6b9KA26kn0FzJfuaeSdw==" 2019-12-10T00:48:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-10T00:48:26.660 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-10T00:48:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": [], 2019-12-10T00:48:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "op_mask": "read, write, delete", 2019-12-10T00:48:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "", 2019-12-10T00:48:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "default_storage_class": "", 2019-12-10T00:48:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_tags": [], 2019-12-10T00:48:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_quota": { 2019-12-10T00:48:26.661 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-10T00:48:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-10T00:48:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-10T00:48:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-10T00:48:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-10T00:48:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-10T00:48:26.662 INFO:teuthology.orchestra.run.smithi067.stdout: "user_quota": { 2019-12-10T00:48:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-10T00:48:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-10T00:48:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-10T00:48:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-10T00:48:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-10T00:48:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-10T00:48:26.663 INFO:teuthology.orchestra.run.smithi067.stdout: "temp_url_keys": [], 2019-12-10T00:48:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "rgw", 2019-12-10T00:48:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: "mfa_ids": [] 2019-12-10T00:48:26.664 INFO:teuthology.orchestra.run.smithi067.stdout:} 2019-12-10T00:48:26.664 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-10T00:48:26.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.730+0000 7fdcd7bccec0 20 remove_watcher() i=0 2019-12-10T00:48:26.731 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.730+0000 7fdcd7bccec0 2 removed watcher, disabling cache 2019-12-10T00:48:26.788 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.786+0000 7fdcd7bccec0 20 remove_watcher() i=1 2019-12-10T00:48:26.845 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.843+0000 7fdcd7bccec0 20 remove_watcher() i=2 2019-12-10T00:48:26.897 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.895+0000 7fdcd7bccec0 20 remove_watcher() i=3 2019-12-10T00:48:26.953 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:26.952+0000 7fdcd7bccec0 20 remove_watcher() i=4 2019-12-10T00:48:27.011 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.010+0000 7fdcd7bccec0 20 remove_watcher() i=5 2019-12-10T00:48:27.068 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.067+0000 7fdcd7bccec0 20 remove_watcher() i=6 2019-12-10T00:48:27.130 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.128+0000 7fdcd7bccec0 20 remove_watcher() i=7 2019-12-10T00:48:27.137 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph user create --subuser bar.client.0:bar --display-name 'Mr. bar.client.0 bar' --secret rsbWq1Nixjla66bUG9miF944C9qVr229GeJx0QQF4KmzQiPVNH82IQ== --email bar.client.0+test@test.test --key-type swift --access full 2019-12-10T00:48:27.264 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.263+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.306 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.305+0000 7fc51f459ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:27.306 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.305+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.314+0000 7fc51f459ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:27.315 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.314+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.318+0000 7fc51f459ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:27.319 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.318+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.328 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.327+0000 7fc51f459ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.493+0000 7fc51f459ec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-10T00:48:27.494 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.493+0000 7fc51f459ec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-10T00:48:27.495 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.493+0000 7fc51f459ec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-10T00:48:27.495 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.493+0000 7fc51f459ec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-10T00:48:27.495 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.493+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.510 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.509+0000 7fc51f459ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:27.510 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.509+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.525+0000 7fc51f459ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:27.530 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.525+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.543 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.542+0000 7fc51f459ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:27.543 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.542+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.569 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.568+0000 7fc51f459ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:27.569 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.568+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.574 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.573+0000 7fc51f459ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.573+0000 7fc51f459ec0 10 cannot find current period zonegroup using local zonegroup 2019-12-10T00:48:27.575 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.573+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.580+0000 7fc51f459ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:27.581 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.580+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.591+0000 7fc51f459ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:27.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.591+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.601+0000 7fc51f459ec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T00:48:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.601+0000 7fc51f459ec0 20 zonegroup default 2019-12-10T00:48:27.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.601+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.606+0000 7fc51f459ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:27.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.606+0000 7fc51f459ec0 10 Cannot find current period zone using local zone 2019-12-10T00:48:27.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.606+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.614+0000 7fc51f459ec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T00:48:27.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.614+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.647+0000 7fc51f459ec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T00:48:27.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.647+0000 7fc51f459ec0 20 rados->read ofs=0 len=0 2019-12-10T00:48:27.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.656+0000 7fc51f459ec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T00:48:27.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.656+0000 7fc51f459ec0 20 zone default found 2019-12-10T00:48:27.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.656+0000 7fc51f459ec0 20 started sync module instance, tier type = 2019-12-10T00:48:27.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:27.656+0000 7fc51f459ec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-10T00:48:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.037+0000 7fc51f459ec0 20 add_watcher() i=0 2019-12-10T00:48:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.037+0000 7fc51f459ec0 20 add_watcher() i=1 2019-12-10T00:48:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.037+0000 7fc51f459ec0 20 add_watcher() i=2 2019-12-10T00:48:28.038 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.037+0000 7fc51f459ec0 20 add_watcher() i=3 2019-12-10T00:48:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.037+0000 7fc51f459ec0 20 add_watcher() i=4 2019-12-10T00:48:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.037+0000 7fc51f459ec0 20 add_watcher() i=5 2019-12-10T00:48:28.039 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.037+0000 7fc51f459ec0 20 add_watcher() i=6 2019-12-10T00:48:28.078 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.077+0000 7fc51f459ec0 20 add_watcher() i=7 2019-12-10T00:48:28.078 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.077+0000 7fc51f459ec0 2 all 8 watchers are set, enabling cache 2019-12-10T00:48:28.078 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.077+0000 7fc4dcfe1700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T00:48:28.083 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.082+0000 7fc50ce84700 0 -- 172.21.15.67:0/1817838115 >> v1:172.21.15.81:6789/0 conn(0x55edfdf8ec60 legacy=0x55edfdf92250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T00:48:28.128 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:28.127+0000 7fc50be82700 0 -- 172.21.15.67:0/2157173854 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x55edfdfa5f70 msgr2=0x55edfdfa8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T00:48:30.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.081+0000 7fc44effd700 20 reqs_thread_entry: start 2019-12-10T00:48:30.083 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.081+0000 7fc44cff9700 20 reqs_thread_entry: start 2019-12-10T00:48:30.083 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.081+0000 7fc51f459ec0 20 init_complete bucket index max shards: 0 2019-12-10T00:48:30.083 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.082+0000 7fc442ffd700 20 reqs_thread_entry: start 2019-12-10T00:48:30.084 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.082+0000 7fc51f459ec0 20 get_system_obj_state: rctx=0x7ffcba67bb18 obj=default.rgw.meta:users.uid:bar.client.0 state=0x55edfe012b30 s->prefetch_data=0 2019-12-10T00:48:30.084 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.082+0000 7fc51f459ec0 10 cache get: name=default.rgw.meta+users.uid+bar.client.0 : miss 2019-12-10T00:48:30.096 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.095+0000 7fc51f459ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x0 2019-12-10T00:48:30.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.095+0000 7fc51f459ec0 10 adding default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-10T00:48:30.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.095+0000 7fc51f459ec0 20 get_system_obj_state: rctx=0x7ffcba67bb28 obj=default.rgw.meta:users.email:bar.client.0+test@test.test state=0x55edfe0148c0 s->prefetch_data=0 2019-12-10T00:48:30.097 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.095+0000 7fc51f459ec0 10 cache get: name=default.rgw.meta+users.email+bar.client.0+test@test.test : miss 2019-12-10T00:48:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.104+0000 7fc51f459ec0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x0 2019-12-10T00:48:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.104+0000 7fc51f459ec0 10 adding default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-10T00:48:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.104+0000 7fc51f459ec0 20 get_system_obj_state: rctx=0x7ffcba67b738 obj=default.rgw.meta:users.swift:bar.client.0:bar state=0x55edfe0148c0 s->prefetch_data=0 2019-12-10T00:48:30.106 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.104+0000 7fc51f459ec0 10 cache get: name=default.rgw.meta+users.swift+bar.client.0:bar : miss 2019-12-10T00:48:30.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.113+0000 7fc51f459ec0 10 cache put: name=default.rgw.meta+users.swift+bar.client.0:bar info.flags=0x0 2019-12-10T00:48:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.113+0000 7fc51f459ec0 10 adding default.rgw.meta+users.swift+bar.client.0:bar to cache LRU end 2019-12-10T00:48:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.113+0000 7fc51f459ec0 20 get_system_obj_state: rctx=0x7ffcba67b8b8 obj=default.rgw.meta:users.swift:bar.client.0:bar state=0x55edfe014b70 s->prefetch_data=0 2019-12-10T00:48:30.115 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.113+0000 7fc51f459ec0 10 cache get: name=default.rgw.meta+users.swift+bar.client.0:bar : type miss (requested=0x6, cached=0x0) 2019-12-10T00:48:30.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.121+0000 7fc51f459ec0 10 cache put: name=default.rgw.meta+users.swift+bar.client.0:bar info.flags=0x0 2019-12-10T00:48:30.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.121+0000 7fc51f459ec0 10 moving default.rgw.meta+users.swift+bar.client.0:bar to cache LRU end 2019-12-10T00:48:30.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.197+0000 7fc51f459ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-10T00:48:30.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.197+0000 7fc51f459ec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-10T00:48:30.198 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.197+0000 7fc51f459ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=612 2019-12-10T00:48:30.213 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.212+0000 7fc4f1ffb700 10 RGWWatcher::handle_notify() notify_id 605590389673 cookie 94480656468336 notifier 4673 bl.length()=612 2019-12-10T00:48:30.214 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.212+0000 7fc4f1ffb700 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-10T00:48:30.214 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:30.212+0000 7fc4f1ffb700 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-10T00:48:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.081+0000 7fc51f459ec0 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2019-12-10T00:48:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.081+0000 7fc51f459ec0 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-10T00:48:31.082 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.081+0000 7fc51f459ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=194 2019-12-10T00:48:31.096 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.094+0000 7fc4f1ffb700 10 RGWWatcher::handle_notify() notify_id 605590389674 cookie 94480656468336 notifier 4673 bl.length()=194 2019-12-10T00:48:31.096 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.094+0000 7fc4f1ffb700 10 cache put: name=default.rgw.meta+users.email+bar.client.0+test@test.test info.flags=0x7 2019-12-10T00:48:31.096 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.094+0000 7fc4f1ffb700 10 moving default.rgw.meta+users.email+bar.client.0+test@test.test to cache LRU end 2019-12-10T00:48:31.421 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.418+0000 7fc51f459ec0 10 cache put: name=default.rgw.meta+users.swift+bar.client.0:bar info.flags=0x7 2019-12-10T00:48:31.421 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.418+0000 7fc51f459ec0 10 moving default.rgw.meta+users.swift+bar.client.0:bar to cache LRU end 2019-12-10T00:48:31.421 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.418+0000 7fc51f459ec0 10 distributing notification oid=default.rgw.control:notify.5 bl.length()=183 2019-12-10T00:48:31.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.430+0000 7fc4f1ffb700 10 RGWWatcher::handle_notify() notify_id 605590388840 cookie 94480656473200 notifier 4673 bl.length()=183 2019-12-10T00:48:31.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.430+0000 7fc4f1ffb700 10 cache put: name=default.rgw.meta+users.swift+bar.client.0:bar info.flags=0x7 2019-12-10T00:48:31.432 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.430+0000 7fc4f1ffb700 10 moving default.rgw.meta+users.swift+bar.client.0:bar to cache LRU end 2019-12-10T00:48:31.991 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.990+0000 7fc51f459ec0 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-10T00:48:31.991 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.990+0000 7fc51f459ec0 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-10T00:48:31.992 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:31.990+0000 7fc51f459ec0 10 distributing notification oid=default.rgw.control:notify.1 bl.length()=636 2019-12-10T00:48:32.002 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.000+0000 7fc4f1ffb700 10 RGWWatcher::handle_notify() notify_id 605590389675 cookie 94480656468336 notifier 4673 bl.length()=636 2019-12-10T00:48:32.002 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.000+0000 7fc4f1ffb700 10 cache put: name=default.rgw.meta+users.uid+bar.client.0 info.flags=0x17 2019-12-10T00:48:32.002 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.000+0000 7fc4f1ffb700 10 moving default.rgw.meta+users.uid+bar.client.0 to cache LRU end 2019-12-10T00:48:32.295 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2019-12-10T00:48:32.296 INFO:teuthology.orchestra.run.smithi067.stdout: "user_id": "bar.client.0", 2019-12-10T00:48:32.296 INFO:teuthology.orchestra.run.smithi067.stdout: "display_name": "Mr. bar.client.0 bar", 2019-12-10T00:48:32.296 INFO:teuthology.orchestra.run.smithi067.stdout: "email": "bar.client.0+test@test.test", 2019-12-10T00:48:32.296 INFO:teuthology.orchestra.run.smithi067.stdout: "suspended": 0, 2019-12-10T00:48:32.296 INFO:teuthology.orchestra.run.smithi067.stdout: "max_buckets": 1000, 2019-12-10T00:48:32.296 INFO:teuthology.orchestra.run.smithi067.stdout: "subusers": [ 2019-12-10T00:48:32.297 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-10T00:48:32.297 INFO:teuthology.orchestra.run.smithi067.stdout: "id": "bar.client.0:bar", 2019-12-10T00:48:32.297 INFO:teuthology.orchestra.run.smithi067.stdout: "permissions": "full-control" 2019-12-10T00:48:32.297 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-10T00:48:32.297 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-10T00:48:32.297 INFO:teuthology.orchestra.run.smithi067.stdout: "keys": [], 2019-12-10T00:48:32.297 INFO:teuthology.orchestra.run.smithi067.stdout: "swift_keys": [ 2019-12-10T00:48:32.298 INFO:teuthology.orchestra.run.smithi067.stdout: { 2019-12-10T00:48:32.298 INFO:teuthology.orchestra.run.smithi067.stdout: "user": "bar.client.0:bar", 2019-12-10T00:48:32.298 INFO:teuthology.orchestra.run.smithi067.stdout: "secret_key": "rsbWq1Nixjla66bUG9miF944C9qVr229GeJx0QQF4KmzQiPVNH82IQ==" 2019-12-10T00:48:32.298 INFO:teuthology.orchestra.run.smithi067.stdout: } 2019-12-10T00:48:32.298 INFO:teuthology.orchestra.run.smithi067.stdout: ], 2019-12-10T00:48:32.298 INFO:teuthology.orchestra.run.smithi067.stdout: "caps": [], 2019-12-10T00:48:32.298 INFO:teuthology.orchestra.run.smithi067.stdout: "op_mask": "read, write, delete", 2019-12-10T00:48:32.298 INFO:teuthology.orchestra.run.smithi067.stdout: "default_placement": "", 2019-12-10T00:48:32.299 INFO:teuthology.orchestra.run.smithi067.stdout: "default_storage_class": "", 2019-12-10T00:48:32.299 INFO:teuthology.orchestra.run.smithi067.stdout: "placement_tags": [], 2019-12-10T00:48:32.299 INFO:teuthology.orchestra.run.smithi067.stdout: "bucket_quota": { 2019-12-10T00:48:32.299 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-10T00:48:32.299 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-10T00:48:32.299 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-10T00:48:32.299 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-10T00:48:32.300 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-10T00:48:32.300 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-10T00:48:32.300 INFO:teuthology.orchestra.run.smithi067.stdout: "user_quota": { 2019-12-10T00:48:32.300 INFO:teuthology.orchestra.run.smithi067.stdout: "enabled": false, 2019-12-10T00:48:32.300 INFO:teuthology.orchestra.run.smithi067.stdout: "check_on_raw": false, 2019-12-10T00:48:32.300 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size": -1, 2019-12-10T00:48:32.300 INFO:teuthology.orchestra.run.smithi067.stdout: "max_size_kb": 0, 2019-12-10T00:48:32.300 INFO:teuthology.orchestra.run.smithi067.stdout: "max_objects": -1 2019-12-10T00:48:32.301 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2019-12-10T00:48:32.301 INFO:teuthology.orchestra.run.smithi067.stdout: "temp_url_keys": [], 2019-12-10T00:48:32.301 INFO:teuthology.orchestra.run.smithi067.stdout: "type": "rgw", 2019-12-10T00:48:32.301 INFO:teuthology.orchestra.run.smithi067.stdout: "mfa_ids": [] 2019-12-10T00:48:32.301 INFO:teuthology.orchestra.run.smithi067.stdout:} 2019-12-10T00:48:32.301 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-10T00:48:32.360 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.359+0000 7fc51f459ec0 20 remove_watcher() i=0 2019-12-10T00:48:32.360 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.359+0000 7fc51f459ec0 2 removed watcher, disabling cache 2019-12-10T00:48:32.468 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.466+0000 7fc51f459ec0 20 remove_watcher() i=1 2019-12-10T00:48:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.520+0000 7fc51f459ec0 20 remove_watcher() i=2 2019-12-10T00:48:32.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.736+0000 7fc51f459ec0 20 remove_watcher() i=3 2019-12-10T00:48:32.884 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.883+0000 7fc51f459ec0 20 remove_watcher() i=4 2019-12-10T00:48:32.930 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.929+0000 7fc51f459ec0 20 remove_watcher() i=5 2019-12-10T00:48:32.983 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:32.981+0000 7fc51f459ec0 20 remove_watcher() i=6 2019-12-10T00:48:33.037 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T00:48:33.036+0000 7fc51f459ec0 20 remove_watcher() i=7 2019-12-10T00:48:33.045 INFO:tasks.swift:Configuring testswift... 2019-12-10T00:48:33.046 INFO:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest/swift && ./bootstrap 2019-12-10T00:48:35.718 INFO:teuthology.orchestra.run.smithi067.stdout:New python executable in /home/ubuntu/cephtest/swift/virtualenv/bin/python 2019-12-10T00:48:37.026 INFO:teuthology.orchestra.run.smithi067.stdout:Installing setuptools, pip, wheel...done. 2019-12-10T00:48:37.367 INFO:teuthology.orchestra.run.smithi067.stderr:Cache entry deserialization failed, entry ignored 2019-12-10T00:48:37.506 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting pip 2019-12-10T00:48:37.595 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl 2019-12-10T00:48:37.662 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: pip 2019-12-10T00:48:37.662 INFO:teuthology.orchestra.run.smithi067.stdout: Found existing installation: pip 9.0.1 2019-12-10T00:48:37.723 INFO:teuthology.orchestra.run.smithi067.stdout: Uninstalling pip-9.0.1: 2019-12-10T00:48:37.767 INFO:teuthology.orchestra.run.smithi067.stdout: Successfully uninstalled pip-9.0.1 2019-12-10T00:48:38.370 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed pip-19.3.1 2019-12-10T00:48:38.661 INFO:teuthology.orchestra.run.smithi067.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-10T00:48:38.853 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:14:59.916 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:38.862 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:14:59.925 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:38.871 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:14:59.935 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:38.877 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:14:59.937 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:38.882 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:15:00.039 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:38.893 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:15:00.050 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:39.016 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:15:00.079 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:39.058 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:15:00.215 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:39.060 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:15:00.124 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:48:39.290 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting setuptools==32.3.1 2019-12-10T00:48:39.294 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/69/19/b1dff551058ce79d88b1e3688f1c735590d7ddf44d10681512133b35019f/setuptools-32.3.1-py2.py3-none-any.whl 2019-12-10T00:48:39.336 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: setuptools 2019-12-10T00:48:39.336 INFO:teuthology.orchestra.run.smithi067.stdout: Found existing installation: setuptools 28.8.0 2019-12-10T00:48:39.351 INFO:teuthology.orchestra.run.smithi067.stdout: Uninstalling setuptools-28.8.0: 2019-12-10T00:48:39.360 INFO:teuthology.orchestra.run.smithi067.stdout: Successfully uninstalled setuptools-28.8.0 2019-12-10T00:48:39.525 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed setuptools-32.3.1 2019-12-10T00:48:39.943 INFO:teuthology.orchestra.run.smithi067.stderr:DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support 2019-12-10T00:48:40.134 INFO:teuthology.orchestra.run.smithi067.stdout:Processing /home/ubuntu/.cache/pip/wheels/54/b7/c7/2ada654ee54483c9329871665aaf4a6056c3ce36f29cf66e67/PyYAML-5.2-cp27-cp27mu-linux_x86_64.whl 2019-12-10T00:48:40.217 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting nose>=1.0.0 2019-12-10T00:48:40.220 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl 2019-12-10T00:48:40.326 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting boto>=2.0b4 2019-12-10T00:48:40.332 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/23/10/c0b78c27298029e4454a472a1919bde20cb182dab1662cec7f2ca1dcc523/boto-2.49.0-py2.py3-none-any.whl 2019-12-10T00:48:40.479 INFO:teuthology.orchestra.run.smithi067.stdout:Processing /home/ubuntu/.cache/pip/wheels/56/0f/19/fbbf81e5764e6d8b74501c4357a88c14c94466ec777c03734c/bunch-1.0.1-cp27-none-any.whl 2019-12-10T00:48:40.734 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting gevent>=0.14 2019-12-10T00:48:40.748 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/e6/e9/3a693414f706e12abe60554cd73c5ae8f848b182ae58018f93d86c9eb418/gevent-1.4.0-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-10T00:48:41.006 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting isodate>=0.4.4 2019-12-10T00:48:41.009 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/9b/9f/b36f7774ff5ea8e428fdcfc4bb332c39ee5b9362ddd3d40d9516a55221b2/isodate-0.6.0-py2.py3-none-any.whl 2019-12-10T00:48:41.097 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting eventlet>=0.9.14 2019-12-10T00:48:41.159 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/ef/01/83454d11bac9015f34e93cf11efcae169390ddf2df97cb73ca86de6465ed/eventlet-0.25.1-py2.py3-none-any.whl (222kB) 2019-12-10T00:48:41.363 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting netifaces>=0.5 2019-12-10T00:48:41.373 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/7e/02/ad1a92a72620cc17d448fe4dbdfbdf8fe1487ee7bfd82bb48308712c2f3c/netifaces-0.10.9-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-10T00:48:41.446 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting xattr<0.7.0,>=0.6 2019-12-10T00:48:41.458 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/c5/80/b25d549ae4bf4f3e9635a331b759ffca2de4dd8a78dc5106d1ca92f5d08d/xattr-0.6.4.tar.gz 2019-12-10T00:48:41.889 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting simplejson>=2.1.3 2019-12-10T00:48:41.900 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/98/87/a7b98aa9256c8843f92878966dc3d8d914c14aad97e2c5ce4798d5743e07/simplejson-3.17.0.tar.gz (83kB) 2019-12-10T00:48:42.373 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting WebOb==1.0 2019-12-10T00:48:42.385 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/bc/eb/91b8dd8f3a8d3c3943bf27a2d322501ad8b227bf29f59e792c2eae092bc9/WebOb-1.0.tar.gz (210kB) 2019-12-10T00:48:42.833 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting greenlet>=0.4.14; platform_python_implementation == "CPython" 2019-12-10T00:48:42.836 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/8b/6e/f2d25875713ad0885c8d3c69269697406652e6f64e1a6bd8264f7a609327/greenlet-0.4.15-cp27-cp27mu-manylinux1_x86_64.whl 2019-12-10T00:48:42.909 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting six 2019-12-10T00:48:42.915 INFO:teuthology.orchestra.run.smithi067.stdout: Using cached https://files.pythonhosted.org/packages/65/26/32b8464df2a97e6dd1b656ed26b2c194606c16fe163c695a992b36c11cdf/six-1.13.0-py2.py3-none-any.whl 2019-12-10T00:48:42.961 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting monotonic>=1.4 2019-12-10T00:48:42.972 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/ac/aa/063eca6a416f397bd99552c534c6d11d57f58f2e94c14780f3bbf818c4cf/monotonic-1.5-py2.py3-none-any.whl 2019-12-10T00:48:43.062 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting dnspython>=1.15.0 2019-12-10T00:48:43.072 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/ec/d3/3aa0e7213ef72b8585747aa0e271a9523e713813b9a20177ebe1e939deb0/dnspython-1.16.0-py2.py3-none-any.whl (188kB) 2019-12-10T00:48:43.222 INFO:teuthology.orchestra.run.smithi067.stdout:Collecting enum34; python_version < "3.4" 2019-12-10T00:48:43.233 INFO:teuthology.orchestra.run.smithi067.stdout: Downloading https://files.pythonhosted.org/packages/c5/db/e56e6b4bbac7c4a06de1c50de6fe1ef3810018ae11732a50f15f62c7d050/enum34-1.1.6-py2-none-any.whl 2019-12-10T00:48:43.267 INFO:teuthology.orchestra.run.smithi067.stdout:Building wheels for collected packages: xattr, simplejson, WebOb 2019-12-10T00:48:43.267 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for xattr (setup.py): started 2019-12-10T00:48:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for xattr (setup.py): finished with status 'done' 2019-12-10T00:48:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: Created wheel for xattr: filename=xattr-0.6.4-cp27-cp27mu-linux_x86_64.whl size=24124 sha256=80c48e285a04f0de12fdef683171784a1a50d38df740ea8b7ad7e4f1dfdf45da 2019-12-10T00:48:43.777 INFO:teuthology.orchestra.run.smithi067.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/cb/2c/40/ea48345a21b521cdacebf61db13e3bf0c42c77ec664e3fe9e5 2019-12-10T00:48:43.783 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for simplejson (setup.py): started 2019-12-10T00:48:44.778 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for simplejson (setup.py): finished with status 'done' 2019-12-10T00:48:44.779 INFO:teuthology.orchestra.run.smithi067.stdout: Created wheel for simplejson: filename=simplejson-3.17.0-cp27-cp27mu-linux_x86_64.whl size=113124 sha256=ef94d62a608b4bacc99a980be7b6a32f498089af94a22c36b33a4deb8d4d0473 2019-12-10T00:48:44.780 INFO:teuthology.orchestra.run.smithi067.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/86/c0/83/dcd0339abb2640544bb8e0938aab2d069cef55e5647ce6e097 2019-12-10T00:48:44.792 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for WebOb (setup.py): started 2019-12-10T00:48:45.058 INFO:teuthology.orchestra.run.smithi067.stdout: Building wheel for WebOb (setup.py): finished with status 'done' 2019-12-10T00:48:45.058 INFO:teuthology.orchestra.run.smithi067.stdout: Created wheel for WebOb: filename=WebOb-1.0-cp27-none-any.whl size=58478 sha256=d89df0608959c84b691c6eed008b80e97993102a789b534dfc236b93e45a77ed 2019-12-10T00:48:45.058 INFO:teuthology.orchestra.run.smithi067.stdout: Stored in directory: /home/ubuntu/.cache/pip/wheels/30/41/c0/8ff9e679d72e05f355e6e2ec87d026131538c953a3b12496eb 2019-12-10T00:48:45.066 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully built xattr simplejson WebOb 2019-12-10T00:48:45.112 INFO:teuthology.orchestra.run.smithi067.stdout:Installing collected packages: PyYAML, nose, boto, bunch, greenlet, gevent, six, isodate, monotonic, dnspython, enum34, eventlet, netifaces, xattr, simplejson, WebOb 2019-12-10T00:48:46.549 INFO:teuthology.orchestra.run.smithi067.stdout:Successfully installed PyYAML-5.2 WebOb-1.0 boto-2.49.0 bunch-1.0.1 dnspython-1.16.0 enum34-1.1.6 eventlet-0.25.1 gevent-1.4.0 greenlet-0.4.15 isodate-0.6.0 monotonic-1.5 netifaces-0.10.9 nose-1.3.7 simplejson-3.17.0 six-1.13.0 xattr-0.6.4 2019-12-10T00:48:46.864 INFO:teuthology.orchestra.run.smithi067.stdout:running develop 2019-12-10T00:48:46.915 INFO:teuthology.orchestra.run.smithi067.stdout:running egg_info 2019-12-10T00:48:46.915 INFO:teuthology.orchestra.run.smithi067.stdout:creating swift.egg-info 2019-12-10T00:48:46.915 INFO:teuthology.orchestra.run.smithi067.stdout:writing swift.egg-info/PKG-INFO 2019-12-10T00:48:46.916 INFO:teuthology.orchestra.run.smithi067.stdout:writing top-level names to swift.egg-info/top_level.txt 2019-12-10T00:48:46.916 INFO:teuthology.orchestra.run.smithi067.stdout:writing dependency_links to swift.egg-info/dependency_links.txt 2019-12-10T00:48:46.928 INFO:teuthology.orchestra.run.smithi067.stdout:writing entry points to swift.egg-info/entry_points.txt 2019-12-10T00:48:46.928 INFO:teuthology.orchestra.run.smithi067.stdout:writing manifest file 'swift.egg-info/SOURCES.txt' 2019-12-10T00:48:46.928 INFO:teuthology.orchestra.run.smithi067.stdout:reading manifest file 'swift.egg-info/SOURCES.txt' 2019-12-10T00:48:46.928 INFO:teuthology.orchestra.run.smithi067.stdout:reading manifest template 'MANIFEST.in' 2019-12-10T00:48:46.929 INFO:teuthology.orchestra.run.smithi067.stdout:writing manifest file 'swift.egg-info/SOURCES.txt' 2019-12-10T00:48:46.929 INFO:teuthology.orchestra.run.smithi067.stdout:running build_ext 2019-12-10T00:48:46.929 INFO:teuthology.orchestra.run.smithi067.stdout:Creating /home/ubuntu/cephtest/swift/virtualenv/lib/python2.7/site-packages/swift.egg-link (link to .) 2019-12-10T00:48:46.929 INFO:teuthology.orchestra.run.smithi067.stdout:Adding swift 1.4.4 to easy-install.pth file 2019-12-10T00:48:46.929 INFO:teuthology.orchestra.run.smithi067.stderr:warning: no files found matching 'ChangeLog' 2019-12-10T00:48:46.944 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.945 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-account-auditor script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.947 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-account-audit script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.948 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-account-reaper script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.949 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-account-replicator script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.950 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-account-server script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.954 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-container-auditor script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.954 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-container-replicator script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.954 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-container-sync script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.954 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-container-server script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.955 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-container-updater script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.955 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-drive-audit script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.956 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-get-nodes script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.958 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-init script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.958 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-object-auditor script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.960 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-object-expirer script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.960 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-object-info script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.961 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-object-replicator script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.962 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-object-server script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.963 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-object-updater script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.964 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-proxy-server script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.968 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-ring-builder script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.969 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-stats-populate script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.981 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-stats-report script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.981 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-dispersion-populate script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.981 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-dispersion-report script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.981 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-bench script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.983 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-recon script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.984 INFO:teuthology.orchestra.run.smithi067.stdout:Installing swift-recon-cron script to /home/ubuntu/cephtest/swift/virtualenv/bin 2019-12-10T00:48:46.984 INFO:teuthology.orchestra.run.smithi067.stdout: 2019-12-10T00:48:46.984 INFO:teuthology.orchestra.run.smithi067.stdout:Installed /home/ubuntu/cephtest/swift 2019-12-10T00:48:46.985 INFO:teuthology.orchestra.run.smithi067.stdout:Processing dependencies for swift==1.4.4 2019-12-10T00:48:46.985 INFO:teuthology.orchestra.run.smithi067.stdout:Finished processing dependencies for swift==1.4.4 2019-12-10T00:48:46.997 INFO:teuthology.orchestra.run.smithi067:> cat > /home/ubuntu/cephtest/archive/testswift.client.0.conf 2019-12-10T00:48:47.048 INFO:teuthology.orchestra.run.smithi067:> SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw' 2019-12-10T00:49:14.019 INFO:teuthology.orchestra.run.smithi067.stderr:testAccountHead (test.functional.tests.TestAccount) ... ok 2019-12-10T00:49:24.433 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerListing (test.functional.tests.TestAccount) ... ok 2019-12-10T00:53:21.948 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerSerializedInfo (test.functional.tests.TestAccount) ... ok 2019-12-10T00:53:33.165 INFO:teuthology.orchestra.run.smithi067.stderr:testContainersOrderedByName (test.functional.tests.TestAccount) ... ok 2019-12-10T00:53:34.631 INFO:teuthology.orchestra.run.smithi067.stderr:testInvalidAuthToken (test.functional.tests.TestAccount) ... ok 2019-12-10T00:53:40.397 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:20:01.460 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:40.421 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:20:01.484 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:40.428 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:20:01.492 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:40.441 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:20:01.505 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:40.487 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:20:01.644 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:40.490 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:20:01.647 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:40.574 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:20:01.637 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:40.575 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:20:01.639 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:40.642 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:20:01.799 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:44.070 INFO:teuthology.orchestra.run.smithi067.stderr:testLastContainerMarker (test.functional.tests.TestAccount) ... ok 2019-12-10T00:53:49.143 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:20:10.207 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:49.153 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:20:10.216 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:49.161 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:20:10.224 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:49.166 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:20:10.229 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:49.170 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:20:10.327 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:49.175 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:20:10.333 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:49.188 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:20:10.252 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:49.205 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:20:10.262 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:49.233 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:20:10.388 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:53:57.333 INFO:teuthology.orchestra.run.smithi067.stderr:testListingLimit (test.functional.tests.TestAccount) ... ok 2019-12-10T00:54:45.902 INFO:teuthology.orchestra.run.smithi067.stderr:testMarkerLimitContainerList (test.functional.tests.TestAccount) ... ok 2019-12-10T00:54:46.280 INFO:teuthology.orchestra.run.smithi067.stderr:testPUT (test.functional.tests.TestAccount) ... ok 2019-12-10T00:58:06.618 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:24:27.660 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:58:06.618 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:24:27.678 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:58:06.658 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:24:27.722 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:58:06.671 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:24:27.734 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:58:06.691 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:24:27.848 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:58:06.696 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:24:27.853 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:58:06.789 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:24:27.852 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:58:06.854 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:24:27.917 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:58:06.861 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:24:28.017 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:01.307 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:25:22.370 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:01.315 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:25:22.379 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:01.325 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:25:22.389 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:01.332 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:25:22.395 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:01.334 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:25:22.492 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:01.344 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:25:22.501 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:01.389 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:25:22.449 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:01.400 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:25:22.461 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:01.467 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:25:22.624 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T00:59:49.543 INFO:teuthology.orchestra.run.smithi067.stderr:testGetRequest (test.functional.tests.TestAccountNoContainers) ... ok 2019-12-10T00:59:51.602 INFO:teuthology.orchestra.run.smithi067.stderr:testGetRequest (test.functional.tests.TestAccountNoContainersUTF8) ... ok 2019-12-10T01:00:17.861 INFO:teuthology.orchestra.run.smithi067.stderr:testAccountHead (test.functional.tests.TestAccountUTF8) ... ok 2019-12-10T01:00:26.579 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerListing (test.functional.tests.TestAccountUTF8) ... ok 2019-12-10T01:02:02.380 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:28:23.443 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:02:02.383 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:28:23.447 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:02:02.430 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:28:23.494 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:02:02.447 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:28:23.510 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:02:02.458 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:28:23.616 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:02:02.462 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:28:23.620 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:02:02.506 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:28:23.569 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:02:02.551 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:28:23.615 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:02:02.556 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:28:23.714 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:11.502 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:30:32.565 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:11.509 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:30:32.572 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:11.528 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:30:32.684 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:11.528 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:30:32.589 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:11.534 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:30:32.692 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:11.535 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:30:32.592 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:11.613 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:30:32.675 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:11.637 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:30:32.701 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:11.684 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:30:32.841 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:04:18.021 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerSerializedInfo (test.functional.tests.TestAccountUTF8) ... ok 2019-12-10T01:04:27.963 INFO:teuthology.orchestra.run.smithi067.stderr:testContainersOrderedByName (test.functional.tests.TestAccountUTF8) ... ok 2019-12-10T01:04:29.777 INFO:teuthology.orchestra.run.smithi067.stderr:testInvalidAuthToken (test.functional.tests.TestAccountUTF8) ... ok 2019-12-10T01:04:41.285 INFO:teuthology.orchestra.run.smithi067.stderr:testLastContainerMarker (test.functional.tests.TestAccountUTF8) ... ok 2019-12-10T01:04:53.889 INFO:teuthology.orchestra.run.smithi067.stderr:testListingLimit (test.functional.tests.TestAccountUTF8) ... ok 2019-12-10T01:06:04.894 INFO:teuthology.orchestra.run.smithi067.stderr:testMarkerLimitContainerList (test.functional.tests.TestAccountUTF8) ... ok 2019-12-10T01:06:04.997 INFO:teuthology.orchestra.run.smithi067.stderr:testPUT (test.functional.tests.TestAccountUTF8) ... ok 2019-12-10T01:09:23.707 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:35:44.770 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:09:23.710 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:35:44.774 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:09:23.718 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:35:44.876 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:09:23.720 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:35:44.783 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:09:23.724 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:35:44.881 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:09:23.727 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:35:44.791 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:09:23.896 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:35:44.960 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:09:23.917 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:35:44.980 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:09:23.918 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:35:45.076 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:10:23.866 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:36:44.929 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:10:23.906 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:36:44.970 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:10:23.950 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:36:45.014 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:10:23.957 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:36:45.021 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:10:23.975 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:36:45.128 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:10:23.975 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:36:45.030 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:10:23.997 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:36:45.137 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:10:24.013 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:36:45.074 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:10:24.061 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:36:45.210 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:11:46.382 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerExistenceCachingProblem (test.functional.tests.TestContainer) ... ok 2019-12-10T01:11:47.879 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerFileList (test.functional.tests.TestContainer) ... ok 2019-12-10T01:11:48.853 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerFileListOnContainerThatDoesNotExist (test.functional.tests.TestContainer) ... ok 2019-12-10T01:11:50.337 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerFileListWithLimit (test.functional.tests.TestContainer) ... ok 2019-12-10T01:11:50.995 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerInfo (test.functional.tests.TestContainer) ... ok 2019-12-10T01:11:51.242 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerInfoOnContainerThatDoesNotExist (test.functional.tests.TestContainer) ... ok 2019-12-10T01:11:59.260 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerNameLimit (test.functional.tests.TestContainer) ... ok 2019-12-10T01:12:03.777 INFO:teuthology.orchestra.run.smithi067.stderr:testCreate (test.functional.tests.TestContainer) ... ok 2019-12-10T01:12:13.889 INFO:teuthology.orchestra.run.smithi067.stderr:testDelete (test.functional.tests.TestContainer) ... ok 2019-12-10T01:12:14.547 INFO:teuthology.orchestra.run.smithi067.stderr:testDeleteOnContainerThatDoesNotExist (test.functional.tests.TestContainer) ... ok 2019-12-10T01:12:19.378 INFO:teuthology.orchestra.run.smithi067.stderr:testDeleteOnContainerWithFiles (test.functional.tests.TestContainer) ... ok 2019-12-10T01:12:19.495 INFO:teuthology.orchestra.run.smithi067.stderr:testFileCreateInContainerThatDoesNotExist (test.functional.tests.TestContainer) ... ok 2019-12-10T01:13:25.330 INFO:teuthology.orchestra.run.smithi067.stderr:testFileListingLimitMarkerPrefix (test.functional.tests.TestContainer) ... ok 2019-12-10T01:13:27.446 INFO:teuthology.orchestra.run.smithi067.stderr:testFileOrder (test.functional.tests.TestContainer) ... ok 2019-12-10T01:13:40.661 INFO:teuthology.orchestra.run.smithi067.stderr:testFileThenContainerDelete (test.functional.tests.TestContainer) ... ok 2019-12-10T01:13:46.295 INFO:teuthology.orchestra.run.smithi067.stderr:testLastFileMarker (test.functional.tests.TestContainer) ... ok 2019-12-10T01:14:10.493 INFO:teuthology.orchestra.run.smithi067.stderr:testMarkerLimitFileList (test.functional.tests.TestContainer) ... ok 2019-12-10T01:14:34.645 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:40:55.709 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:14:34.649 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:40:55.712 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:14:34.661 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:40:55.724 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:14:34.665 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:40:55.728 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:14:34.673 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:40:55.831 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:14:34.692 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:40:55.850 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:14:34.810 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:40:55.874 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:14:34.829 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:40:55.892 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:14:34.912 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:40:56.070 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:14:52.869 INFO:teuthology.orchestra.run.smithi067.stderr:testPrefixAndLimit (test.functional.tests.TestContainer) ... ok 2019-12-10T01:14:52.897 INFO:teuthology.orchestra.run.smithi067.stderr:testTooLongName (test.functional.tests.TestContainer) ... ok 2019-12-10T01:15:45.938 INFO:teuthology.orchestra.run.smithi067.stderr:testUnorderedList (test.functional.tests.TestContainer) ... ok 2019-12-10T01:18:45.566 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:45:06.630 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:18:45.586 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:45:06.649 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:18:45.597 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:45:06.661 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:18:45.617 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:45:06.680 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:18:45.639 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:45:06.798 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:18:45.645 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:45:06.804 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:18:45.723 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:45:06.786 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:18:45.762 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:45:06.921 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:18:45.789 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:45:06.853 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:18:59.017 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerListing (test.functional.tests.TestContainerPaths) ... ok 2019-12-10T01:19:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:testStructure (test.functional.tests.TestContainerPaths) ... ok 2019-12-10T01:19:21.334 INFO:teuthology.orchestra.run.smithi067.stderr:testTraverseContainer (test.functional.tests.TestContainerPaths) ... ok 2019-12-10T01:19:44.959 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:46:06.020 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:19:44.962 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:46:06.025 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:19:44.975 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:46:06.038 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:19:44.980 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:46:06.043 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:19:44.991 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:46:06.148 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:19:44.999 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:46:06.158 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:19:45.031 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:46:06.095 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:19:45.035 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:46:06.099 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:19:45.095 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:46:06.253 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:20:34.816 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerExistenceCachingProblem (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:20:35.988 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerFileList (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:20:37.150 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerFileListOnContainerThatDoesNotExist (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:20:38.538 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerFileListWithLimit (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:20:39.115 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerInfo (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:20:39.365 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerInfoOnContainerThatDoesNotExist (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:20:47.122 INFO:teuthology.orchestra.run.smithi067.stderr:testContainerNameLimit (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:20:52.078 INFO:teuthology.orchestra.run.smithi067.stderr:testCreate (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:21:01.923 INFO:teuthology.orchestra.run.smithi067.stderr:testDelete (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:21:02.046 INFO:teuthology.orchestra.run.smithi067.stderr:testDeleteOnContainerThatDoesNotExist (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:21:07.522 INFO:teuthology.orchestra.run.smithi067.stderr:testDeleteOnContainerWithFiles (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:21:07.700 INFO:teuthology.orchestra.run.smithi067.stderr:testFileCreateInContainerThatDoesNotExist (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:22:08.719 INFO:teuthology.orchestra.run.smithi067.stderr:testFileListingLimitMarkerPrefix (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:22:10.528 INFO:teuthology.orchestra.run.smithi067.stderr:testFileOrder (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:22:24.859 INFO:teuthology.orchestra.run.smithi067.stderr:testFileThenContainerDelete (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:22:27.970 INFO:teuthology.orchestra.run.smithi067.stderr:testLastFileMarker (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:22:49.333 INFO:teuthology.orchestra.run.smithi067.stderr:testMarkerLimitFileList (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:23:29.919 INFO:teuthology.orchestra.run.smithi067.stderr:testPrefixAndLimit (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:23:29.951 INFO:teuthology.orchestra.run.smithi067.stderr:testTooLongName (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:24:20.879 INFO:teuthology.orchestra.run.smithi067.stderr:testUnorderedList (test.functional.tests.TestContainerUTF8) ... ok 2019-12-10T01:24:56.671 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:51:17.733 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:24:56.681 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:51:17.742 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:24:56.694 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:51:17.757 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:24:56.703 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:51:17.766 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:24:56.704 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:51:17.861 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:24:56.713 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:51:17.871 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:24:56.897 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:51:17.960 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:24:56.897 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:51:17.960 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:24:56.986 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:51:18.144 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:07.041 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:53:28.104 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:07.060 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:53:28.123 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:07.075 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:53:28.234 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:07.076 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:53:28.139 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:07.109 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:53:28.265 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:07.115 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:53:28.178 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:07.139 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:53:28.202 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:07.190 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:53:28.253 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:07.194 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:53:28.353 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:27:17.956 INFO:teuthology.orchestra.run.smithi067.stderr:test_get_manifest (test.functional.tests.TestDlo) ... ok 2019-12-10T01:27:18.202 INFO:teuthology.orchestra.run.smithi067.stderr:test_get_manifest_document_itself (test.functional.tests.TestDlo) ... ok 2019-12-10T01:27:22.819 INFO:teuthology.orchestra.run.smithi067.stderr:test_get_range (test.functional.tests.TestDlo) ... ok 2019-12-10T01:27:24.060 INFO:teuthology.orchestra.run.smithi067.stderr:test_get_range_out_of_range (test.functional.tests.TestDlo) ... ok 2019-12-10T01:28:15.088 INFO:teuthology.orchestra.run.smithi067.stderr:test_get_manifest (test.functional.tests.TestDloUTF8) ... ok 2019-12-10T01:28:15.269 INFO:teuthology.orchestra.run.smithi067.stderr:test_get_manifest_document_itself (test.functional.tests.TestDloUTF8) ... ok 2019-12-10T01:28:19.213 INFO:teuthology.orchestra.run.smithi067.stderr:test_get_range (test.functional.tests.TestDloUTF8) ... ok 2019-12-10T01:28:20.342 INFO:teuthology.orchestra.run.smithi067.stderr:test_get_range_out_of_range (test.functional.tests.TestDloUTF8) ... ok 2019-12-10T01:28:49.897 INFO:teuthology.orchestra.run.smithi067.stderr:testBadHeaders (test.functional.tests.TestFile) ... ok 2019-12-10T01:28:49.914 INFO:teuthology.orchestra.run.smithi067.stderr:testBlankMetadataName (test.functional.tests.TestFile) ... ok 2019-12-10T01:30:07.818 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:56:28.881 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:30:07.823 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:56:28.887 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:30:07.835 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:56:28.898 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:30:07.841 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:56:28.904 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:30:07.844 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:56:29.001 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:30:07.847 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:56:29.006 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:30:07.984 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:04:56:29.046 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:30:08.014 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:04:56:29.072 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:30:08.014 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:04:56:29.164 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:18.812 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:01:39.876 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:18.818 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:01:39.881 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:18.827 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:01:39.891 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:18.831 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:01:39.894 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:18.836 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:01:39.995 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:18.852 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:01:40.010 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:19.015 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:01:40.078 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:19.019 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:01:40.083 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:19.074 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:01:40.233 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:28.743 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:01:49.805 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:28.778 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:01:49.841 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:28.792 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:01:49.855 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:28.803 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:01:49.866 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:28.828 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:01:49.892 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:28.846 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:01:49.910 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:28.856 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:01:50.015 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:28.864 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:01:50.023 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:35:28.965 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:01:50.120 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:36:45.534 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:36:45.519+0000 1694f700 -1 osd.4 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.102882+0000 front 2019-12-10T01:36:01.979186+0000 (oldest deadline 2019-12-10T01:36:45.375717+0000) 2019-12-10T01:36:46.512 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:36:46.505+0000 1694f700 -1 osd.4 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.102882+0000 front 2019-12-10T01:36:01.979186+0000 (oldest deadline 2019-12-10T01:36:45.375717+0000) 2019-12-10T01:36:46.640 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:36:46.630+0000 1694f700 -1 osd.1 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.751627+0000 front 2019-12-10T01:36:02.780297+0000 (oldest deadline 2019-12-10T01:36:46.155748+0000) 2019-12-10T01:36:46.724 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:36:46.710+0000 1694f700 -1 osd.6 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.634612+0000 front 2019-12-10T01:36:02.706062+0000 (oldest deadline 2019-12-10T01:36:46.160766+0000) 2019-12-10T01:36:47.324 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:36:47.317+0000 1694f700 -1 osd.2 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:01.981674+0000 front 2019-12-10T01:36:01.288856+0000 (oldest deadline 2019-12-10T01:36:46.491576+0000) 2019-12-10T01:36:47.547 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:36:47.541+0000 1694f700 -1 osd.4 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.102882+0000 front 2019-12-10T01:36:01.979186+0000 (oldest deadline 2019-12-10T01:36:45.375717+0000) 2019-12-10T01:36:47.549 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:36:47.534+0000 1694f700 -1 osd.5 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.072609+0000 front 2019-12-10T01:36:02.690942+0000 (oldest deadline 2019-12-10T01:36:47.289497+0000) 2019-12-10T01:36:47.602 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:36:47.599+0000 1694f700 -1 osd.1 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.751627+0000 front 2019-12-10T01:36:02.780297+0000 (oldest deadline 2019-12-10T01:36:46.155748+0000) 2019-12-10T01:36:47.730 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:36:47.718+0000 1694f700 -1 osd.6 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.634612+0000 front 2019-12-10T01:36:02.706062+0000 (oldest deadline 2019-12-10T01:36:46.160766+0000) 2019-12-10T01:36:48.362 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:36:48.356+0000 1694f700 -1 osd.2 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:01.981674+0000 front 2019-12-10T01:36:01.288856+0000 (oldest deadline 2019-12-10T01:36:46.491576+0000) 2019-12-10T01:36:48.589 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:36:48.557+0000 1694f700 -1 osd.4 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.102882+0000 front 2019-12-10T01:36:01.979186+0000 (oldest deadline 2019-12-10T01:36:45.375717+0000) 2019-12-10T01:36:48.596 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:36:48.587+0000 1694f700 -1 osd.5 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.072609+0000 front 2019-12-10T01:36:02.690942+0000 (oldest deadline 2019-12-10T01:36:47.289497+0000) 2019-12-10T01:36:48.641 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:36:48.639+0000 1694f700 -1 osd.1 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.751627+0000 front 2019-12-10T01:36:02.780297+0000 (oldest deadline 2019-12-10T01:36:46.155748+0000) 2019-12-10T01:36:48.757 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:36:48.732+0000 1694f700 -1 osd.6 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.634612+0000 front 2019-12-10T01:36:02.706062+0000 (oldest deadline 2019-12-10T01:36:46.160766+0000) 2019-12-10T01:36:49.015 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:36:49.007+0000 1694f700 -1 osd.0 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:06.315828+0000 front 2019-12-10T01:36:04.311789+0000 (oldest deadline 2019-12-10T01:36:48.960016+0000) 2019-12-10T01:36:49.384 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:36:49.381+0000 1694f700 -1 osd.2 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:01.981674+0000 front 2019-12-10T01:36:01.288856+0000 (oldest deadline 2019-12-10T01:36:46.491576+0000) 2019-12-10T01:36:49.617 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:36:49.609+0000 1694f700 -1 osd.4 141 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:36:02.102882+0000 front 2019-12-10T01:36:01.979186+0000 (oldest deadline 2019-12-10T01:36:45.375717+0000) 2019-12-10T01:37:39.438 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:04:00.502 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:37:39.484 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:04:00.548 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:37:39.488 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:04:00.549 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:37:39.518 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:04:00.581 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:37:39.573 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:04:00.731 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:37:39.576 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:04:00.734 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:37:39.616 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:04:00.679 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:37:39.665 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:04:00.725 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:37:39.688 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:04:00.846 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:37:47.945 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:37:47.934+0000 1694f700 -1 osd.6 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:05.275526+0000 front 2019-12-10T01:37:05.358674+0000 (oldest deadline 2019-12-10T01:37:47.518471+0000) 2019-12-10T01:37:48.721 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:37:48.714+0000 1694f700 -1 osd.2 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:02.400100+0000 front 2019-12-10T01:37:02.488910+0000 (oldest deadline 2019-12-10T01:37:48.082671+0000) 2019-12-10T01:37:48.948 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:37:48.939+0000 1694f700 -1 osd.6 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:05.275526+0000 front 2019-12-10T01:37:05.358674+0000 (oldest deadline 2019-12-10T01:37:47.518471+0000) 2019-12-10T01:37:49.672 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:37:49.669+0000 1694f700 -1 osd.2 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:02.400100+0000 front 2019-12-10T01:37:02.488910+0000 (oldest deadline 2019-12-10T01:37:48.082671+0000) 2019-12-10T01:37:49.970 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:37:49.964+0000 1694f700 -1 osd.6 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:05.275526+0000 front 2019-12-10T01:37:05.358674+0000 (oldest deadline 2019-12-10T01:37:47.518471+0000) 2019-12-10T01:37:50.233 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:37:50.230+0000 1694f700 -1 osd.0 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:04.333088+0000 front 2019-12-10T01:37:04.243190+0000 (oldest deadline 2019-12-10T01:37:49.524872+0000) 2019-12-10T01:37:50.678 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:37:50.660+0000 1694f700 -1 osd.2 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:02.400100+0000 front 2019-12-10T01:37:02.488910+0000 (oldest deadline 2019-12-10T01:37:48.082671+0000) 2019-12-10T01:37:50.780 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:37:50.774+0000 1694f700 -1 osd.4 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:06.104083+0000 front 2019-12-10T01:37:06.228242+0000 (oldest deadline 2019-12-10T01:37:50.159647+0000) 2019-12-10T01:37:50.880 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:37:50.869+0000 1694f700 -1 osd.7 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:06.587455+0000 front 2019-12-10T01:37:06.674413+0000 (oldest deadline 2019-12-10T01:37:50.051405+0000) 2019-12-10T01:37:51.004 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:37:50.995+0000 1694f700 -1 osd.6 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:05.275526+0000 front 2019-12-10T01:37:05.358674+0000 (oldest deadline 2019-12-10T01:37:47.518471+0000) 2019-12-10T01:37:51.258 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:37:51.254+0000 1694f700 -1 osd.0 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:04.333088+0000 front 2019-12-10T01:37:04.243190+0000 (oldest deadline 2019-12-10T01:37:49.524872+0000) 2019-12-10T01:37:51.651 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:37:51.648+0000 1694f700 -1 osd.2 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:02.400100+0000 front 2019-12-10T01:37:02.488910+0000 (oldest deadline 2019-12-10T01:37:48.082671+0000) 2019-12-10T01:37:51.814 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:37:51.808+0000 1694f700 -1 osd.4 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:06.104083+0000 front 2019-12-10T01:37:06.228242+0000 (oldest deadline 2019-12-10T01:37:50.159647+0000) 2019-12-10T01:37:51.842 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:37:51.836+0000 1694f700 -1 osd.5 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:04.477261+0000 front 2019-12-10T01:37:04.478989+0000 (oldest deadline 2019-12-10T01:37:51.265762+0000) 2019-12-10T01:37:51.897 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:37:51.892+0000 1694f700 -1 osd.7 145 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:37:06.587455+0000 front 2019-12-10T01:37:06.674413+0000 (oldest deadline 2019-12-10T01:37:50.051405+0000) 2019-12-10T01:38:02.053 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:38:01.941+0000 124b0700 -1 osd.3 150 build_incremental_map_msg missing incremental map 150 2019-12-10T01:38:02.108 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:38:01.975+0000 124b0700 -1 osd.3 150 build_incremental_map_msg missing incremental map 150 2019-12-10T01:38:02.161 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:38:02.024+0000 36790700 -1 osd.3 150 build_incremental_map_msg missing incremental map 150 2019-12-10T01:38:02.262 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:38:02.073+0000 38794700 -1 osd.3 150 build_incremental_map_msg missing incremental map 150 2019-12-10T01:38:02.264 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:38:02.084+0000 11caf700 -1 osd.3 150 build_incremental_map_msg missing incremental map 150 2019-12-10T01:38:02.325 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:38:02.182+0000 31f87700 -1 osd.3 150 build_incremental_map_msg missing incremental map 150 2019-12-10T01:38:02.342 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:38:02.202+0000 31f87700 -1 osd.3 150 build_incremental_map_msg missing incremental map 150 2019-12-10T01:38:02.361 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:38:02.265+0000 36790700 -1 osd.3 150 build_incremental_map_msg missing incremental map 150 2019-12-10T01:38:22.551 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:38:22.547+0000 1694f700 -1 osd.0 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103241 5.5 5:a93a5511:::notify.2:head [watch reconnect cookie 821342224 gen 69] snapc 0=[] ondisk+write+known_if_redirected e146) 2019-12-10T01:38:36.904 INFO:teuthology.orchestra.run.smithi067.stderr:testChunkedPut (test.functional.tests.TestFile) ... ok 2019-12-10T01:38:44.287 INFO:teuthology.orchestra.run.smithi067.stderr:testContentTypeGuessing (test.functional.tests.TestFile) ... ok 2019-12-10T01:39:01.752 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:39:01.750+0000 37f93700 -1 osd.3 151 build_incremental_map_msg missing incremental map 150 2019-12-10T01:39:02.243 INFO:teuthology.orchestra.run.smithi067.stderr:testCopy (test.functional.tests.TestFile) ... ok 2019-12-10T01:39:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:testCopy404s (test.functional.tests.TestFile) ... ok 2019-12-10T01:39:10.757 INFO:teuthology.orchestra.run.smithi067.stderr:testCopyDestinationSlashProblems (test.functional.tests.TestFile) ... ok 2019-12-10T01:39:28.881 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:05:49.944 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:39:28.892 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:05:49.949 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:39:28.917 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:05:50.074 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:39:28.921 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:05:50.079 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:39:28.998 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:05:50.061 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:39:29.007 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:05:50.070 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:39:29.100 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:05:50.164 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:39:29.194 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:05:50.353 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:39:29.206 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:05:50.269 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:39:33.140 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-10T01:39:33.071+0000 20e25700 -1 RGWWatcher::handle_error cookie 821283136 err (107) Transport endpoint is not connected 2019-12-10T01:39:34.585 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:39:34.484+0000 1694f700 -1 osd.3 154 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected e152) 2019-12-10T01:39:35.638 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:39:35.612+0000 1694f700 -1 osd.3 154 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected e152) 2019-12-10T01:39:36.669 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:39:36.643+0000 1694f700 -1 osd.3 154 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected e152) 2019-12-10T01:39:37.729 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:39:37.696+0000 1694f700 -1 osd.3 154 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7:4f78e9a2:gc::gc.8:head [call rgw_gc.rgw_gc_queue_list_entries in=46b] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected e152) 2019-12-10T01:39:52.382 INFO:teuthology.orchestra.run.smithi067.stderr:testCopyFromHeader (test.functional.tests.TestFile) ... ok 2019-12-10T01:40:26.911 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:26.902+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:27.917 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:27.908+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:28.923 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:28.917+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:29.888 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:29.883+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:30.905 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:30.898+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:30.908 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:30.904+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:30.928 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:30.922+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:31.234 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:40:31.232+0000 1694f700 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:46.315642+0000 front 2019-12-10T01:39:46.317443+0000 (oldest deadline 2019-12-10T01:40:31.225119+0000) 2019-12-10T01:40:31.924 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:31.917+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:31.940 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:31.933+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:31.941 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:31.937+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:32.115 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:40:32.101+0000 1694f700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.156414+0000 front 2019-12-10T01:39:49.327203+0000 (oldest deadline 2019-12-10T01:40:31.863319+0000) 2019-12-10T01:40:32.265 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:40:32.260+0000 1694f700 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:46.315642+0000 front 2019-12-10T01:39:46.317443+0000 (oldest deadline 2019-12-10T01:40:31.225119+0000) 2019-12-10T01:40:32.889 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:32.877+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:32.964 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:32.958+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:32.967 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:32.963+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:33.173 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:40:33.166+0000 1694f700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.156414+0000 front 2019-12-10T01:39:49.327203+0000 (oldest deadline 2019-12-10T01:40:31.863319+0000) 2019-12-10T01:40:33.271 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:40:33.267+0000 1694f700 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:46.315642+0000 front 2019-12-10T01:39:46.317443+0000 (oldest deadline 2019-12-10T01:40:31.225119+0000) 2019-12-10T01:40:33.341 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:40:33.335+0000 1694f700 -1 osd.1 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:48.033316+0000 front 2019-12-10T01:39:48.075204+0000 (oldest deadline 2019-12-10T01:40:32.933917+0000) 2019-12-10T01:40:33.604 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:40:33.598+0000 1694f700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.565889+0000 front 2019-12-10T01:39:49.609600+0000 (oldest deadline 2019-12-10T01:40:33.575483+0000) 2019-12-10T01:40:33.879 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:33.873+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:33.956 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:33.949+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:33.956 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:33.951+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:34.128 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:40:34.121+0000 1694f700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.156414+0000 front 2019-12-10T01:39:49.327203+0000 (oldest deadline 2019-12-10T01:40:31.863319+0000) 2019-12-10T01:40:34.295 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:40:34.292+0000 1694f700 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:46.315642+0000 front 2019-12-10T01:39:46.317443+0000 (oldest deadline 2019-12-10T01:40:31.225119+0000) 2019-12-10T01:40:34.351 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:40:34.348+0000 1694f700 -1 osd.1 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:48.033316+0000 front 2019-12-10T01:39:48.075204+0000 (oldest deadline 2019-12-10T01:40:32.933917+0000) 2019-12-10T01:40:34.658 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:40:34.643+0000 1694f700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.565889+0000 front 2019-12-10T01:39:49.609600+0000 (oldest deadline 2019-12-10T01:40:33.575483+0000) 2019-12-10T01:40:34.932 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:34.925+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:34.983 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:34.978+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:34.984 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:34.980+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:35.165 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:40:35.158+0000 1694f700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.156414+0000 front 2019-12-10T01:39:49.327203+0000 (oldest deadline 2019-12-10T01:40:31.863319+0000) 2019-12-10T01:40:35.325 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:40:35.322+0000 1694f700 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:46.315642+0000 front 2019-12-10T01:39:46.317443+0000 (oldest deadline 2019-12-10T01:40:31.225119+0000) 2019-12-10T01:40:35.376 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:40:35.366+0000 1694f700 -1 osd.0 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.718156+0000 front 2019-12-10T01:39:49.904540+0000 (oldest deadline 2019-12-10T01:40:34.403766+0000) 2019-12-10T01:40:35.393 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:40:35.386+0000 1694f700 -1 osd.1 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:48.033316+0000 front 2019-12-10T01:39:48.075204+0000 (oldest deadline 2019-12-10T01:40:32.933917+0000) 2019-12-10T01:40:35.667 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:40:35.662+0000 1694f700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.565889+0000 front 2019-12-10T01:39:49.609600+0000 (oldest deadline 2019-12-10T01:40:33.575483+0000) 2019-12-10T01:40:35.949 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:35.942+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:36.033 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:36.022+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:36.033 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:36.024+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:36.183 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:40:36.171+0000 1694f700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.156414+0000 front 2019-12-10T01:39:49.327203+0000 (oldest deadline 2019-12-10T01:40:31.863319+0000) 2019-12-10T01:40:36.366 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:40:36.358+0000 1694f700 -1 osd.1 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:48.033316+0000 front 2019-12-10T01:39:48.075204+0000 (oldest deadline 2019-12-10T01:40:32.933917+0000) 2019-12-10T01:40:36.374 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:40:36.371+0000 1694f700 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:46.315642+0000 front 2019-12-10T01:39:46.317443+0000 (oldest deadline 2019-12-10T01:40:31.225119+0000) 2019-12-10T01:40:36.386 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:40:36.383+0000 1694f700 -1 osd.0 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.718156+0000 front 2019-12-10T01:39:49.904540+0000 (oldest deadline 2019-12-10T01:40:34.403766+0000) 2019-12-10T01:40:36.700 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:40:36.694+0000 1694f700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.565889+0000 front 2019-12-10T01:39:49.609600+0000 (oldest deadline 2019-12-10T01:40:33.575483+0000) 2019-12-10T01:40:36.921 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:36.915+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:37.053 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:37.046+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:37.054 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:37.048+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:37.186 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:40:37.179+0000 1694f700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.156414+0000 front 2019-12-10T01:39:49.327203+0000 (oldest deadline 2019-12-10T01:40:31.863319+0000) 2019-12-10T01:40:37.347 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:40:37.343+0000 1694f700 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:46.315642+0000 front 2019-12-10T01:39:46.317443+0000 (oldest deadline 2019-12-10T01:40:31.225119+0000) 2019-12-10T01:40:37.396 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:40:37.392+0000 1694f700 -1 osd.1 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:48.033316+0000 front 2019-12-10T01:39:48.075204+0000 (oldest deadline 2019-12-10T01:40:32.933917+0000) 2019-12-10T01:40:37.433 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:40:37.430+0000 1694f700 -1 osd.0 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.718156+0000 front 2019-12-10T01:39:49.904540+0000 (oldest deadline 2019-12-10T01:40:34.403766+0000) 2019-12-10T01:40:37.691 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:40:37.686+0000 1694f700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.565889+0000 front 2019-12-10T01:39:49.609600+0000 (oldest deadline 2019-12-10T01:40:33.575483+0000) 2019-12-10T01:40:37.889 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:37.883+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:38.036 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:38.029+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:38.038 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:38.033+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:38.206 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:40:38.199+0000 1694f700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.156414+0000 front 2019-12-10T01:39:49.327203+0000 (oldest deadline 2019-12-10T01:40:31.863319+0000) 2019-12-10T01:40:38.356 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:40:38.353+0000 1694f700 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:46.315642+0000 front 2019-12-10T01:39:46.317443+0000 (oldest deadline 2019-12-10T01:40:31.225119+0000) 2019-12-10T01:40:38.402 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:40:38.399+0000 1694f700 -1 osd.1 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:48.033316+0000 front 2019-12-10T01:39:48.075204+0000 (oldest deadline 2019-12-10T01:40:32.933917+0000) 2019-12-10T01:40:38.430 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:40:38.427+0000 1694f700 -1 osd.0 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.718156+0000 front 2019-12-10T01:39:49.904540+0000 (oldest deadline 2019-12-10T01:40:34.403766+0000) 2019-12-10T01:40:38.745 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:40:38.732+0000 1694f700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.565889+0000 front 2019-12-10T01:39:49.609600+0000 (oldest deadline 2019-12-10T01:40:33.575483+0000) 2019-12-10T01:40:38.913 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:38.904+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:39.041 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:39.034+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:39.041 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:39.036+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:39.226 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:40:39.213+0000 1694f700 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.156414+0000 front 2019-12-10T01:39:49.327203+0000 (oldest deadline 2019-12-10T01:40:31.863319+0000) 2019-12-10T01:40:39.405 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:40:39.400+0000 1694f700 -1 osd.2 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:46.315642+0000 front 2019-12-10T01:39:46.317443+0000 (oldest deadline 2019-12-10T01:40:31.225119+0000) 2019-12-10T01:40:39.422 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:40:39.419+0000 1694f700 -1 osd.1 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:48.033316+0000 front 2019-12-10T01:39:48.075204+0000 (oldest deadline 2019-12-10T01:40:32.933917+0000) 2019-12-10T01:40:39.433 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:40:39.429+0000 1694f700 -1 osd.0 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.718156+0000 front 2019-12-10T01:39:49.904540+0000 (oldest deadline 2019-12-10T01:40:34.403766+0000) 2019-12-10T01:40:39.704 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:40:39.697+0000 1694f700 -1 osd.7 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.565889+0000 front 2019-12-10T01:39:49.609600+0000 (oldest deadline 2019-12-10T01:40:33.575483+0000) 2019-12-10T01:40:39.964 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:40:39.955+0000 1694f700 -1 osd.6 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.587672+0000 front 2019-12-10T01:39:49.631413+0000 (oldest deadline 2019-12-10T01:40:30.654206+0000) 2019-12-10T01:40:40.034 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:40.028+0000 1694f700 -1 osd.4 154 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:39:49.207067+0000 front 2019-12-10T01:39:49.327307+0000 (oldest deadline 2019-12-10T01:40:30.878411+0000) 2019-12-10T01:40:40.034 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:40.030+0000 1694f700 -1 osd.4 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:40:41.343 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:40:41.196+0000 1694f700 -1 osd.4 155 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103695 6.2 6:4ac63539:root::4e1b4f857ff4476da1eff32940544b03:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e154) 2019-12-10T01:41:00.147 INFO:teuthology.orchestra.run.smithi067.stderr:testCopyFromHeader404s (test.functional.tests.TestFile) ... ok 2019-12-10T01:41:00.983 INFO:teuthology.orchestra.run.smithi067.stderr:testCopyNoDestinationHeader (test.functional.tests.TestFile) ... ok 2019-12-10T01:41:05.575 INFO:teuthology.orchestra.run.smithi067.stderr:testDelete (test.functional.tests.TestFile) ... ok 2019-12-10T01:41:06.380 INFO:teuthology.orchestra.run.smithi067.stderr:testDeleteOfFileThatDoesNotExist (test.functional.tests.TestFile) ... ok 2019-12-10T01:41:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:testDeleteThen404s (test.functional.tests.TestFile) ... ok 2019-12-10T01:41:10.642 INFO:teuthology.orchestra.run.smithi067.stderr:testEtagResponse (test.functional.tests.TestFile) ... ok 2019-12-10T01:41:10.694 INFO:teuthology.orchestra.run.smithi067.stderr:testEtagWayoff (test.functional.tests.TestFile) ... ok 2019-12-10T01:41:38.302 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:41:38.296+0000 1694f700 -1 osd.4 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.448064+0000 front 2019-12-10T01:40:57.471258+0000 (oldest deadline 2019-12-10T01:41:37.879942+0000) 2019-12-10T01:41:39.302 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:41:39.296+0000 1694f700 -1 osd.4 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.448064+0000 front 2019-12-10T01:40:57.471258+0000 (oldest deadline 2019-12-10T01:41:37.879942+0000) 2019-12-10T01:41:39.876 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:41:39.870+0000 1694f700 -1 osd.5 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.396784+0000 front 2019-12-10T01:40:57.398288+0000 (oldest deadline 2019-12-10T01:41:39.196323+0000) 2019-12-10T01:41:40.204 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:40.201+0000 1694f700 -1 osd.0 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.470601+0000 front 2019-12-10T01:40:55.498885+0000 (oldest deadline 2019-12-10T01:41:40.154554+0000) 2019-12-10T01:41:40.312 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:41:40.306+0000 1694f700 -1 osd.4 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.448064+0000 front 2019-12-10T01:40:57.471258+0000 (oldest deadline 2019-12-10T01:41:37.879942+0000) 2019-12-10T01:41:40.380 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:41:40.375+0000 1694f700 -1 osd.7 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.445847+0000 front 2019-12-10T01:40:55.556959+0000 (oldest deadline 2019-12-10T01:41:39.401025+0000) 2019-12-10T01:41:40.505 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:41:40.499+0000 1694f700 -1 osd.6 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.546294+0000 front 2019-12-10T01:40:58.047498+0000 (oldest deadline 2019-12-10T01:41:39.762783+0000) 2019-12-10T01:41:40.878 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:41:40.871+0000 1694f700 -1 osd.5 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.396784+0000 front 2019-12-10T01:40:57.398288+0000 (oldest deadline 2019-12-10T01:41:39.196323+0000) 2019-12-10T01:41:41.107 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:41:41.103+0000 1694f700 -1 osd.2 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.046867+0000 front 2019-12-10T01:40:55.114340+0000 (oldest deadline 2019-12-10T01:41:40.884657+0000) 2019-12-10T01:41:41.205 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:41.200+0000 1694f700 -1 osd.0 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.470601+0000 front 2019-12-10T01:40:55.498885+0000 (oldest deadline 2019-12-10T01:41:40.154554+0000) 2019-12-10T01:41:41.306 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:41:41.296+0000 1694f700 -1 osd.4 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.448064+0000 front 2019-12-10T01:40:57.471258+0000 (oldest deadline 2019-12-10T01:41:37.879942+0000) 2019-12-10T01:41:41.400 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:41:41.395+0000 1694f700 -1 osd.7 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.445847+0000 front 2019-12-10T01:40:55.556959+0000 (oldest deadline 2019-12-10T01:41:39.401025+0000) 2019-12-10T01:41:41.480 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:41:41.475+0000 1694f700 -1 osd.6 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.546294+0000 front 2019-12-10T01:40:58.047498+0000 (oldest deadline 2019-12-10T01:41:39.762783+0000) 2019-12-10T01:41:41.915 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:41:41.909+0000 1694f700 -1 osd.5 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.396784+0000 front 2019-12-10T01:40:57.398288+0000 (oldest deadline 2019-12-10T01:41:39.196323+0000) 2019-12-10T01:41:42.121 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:41:42.117+0000 1694f700 -1 osd.2 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.046867+0000 front 2019-12-10T01:40:55.114340+0000 (oldest deadline 2019-12-10T01:41:40.884657+0000) 2019-12-10T01:41:42.236 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:42.232+0000 1694f700 -1 osd.0 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.470601+0000 front 2019-12-10T01:40:55.498885+0000 (oldest deadline 2019-12-10T01:41:40.154554+0000) 2019-12-10T01:41:42.323 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:41:42.317+0000 1694f700 -1 osd.4 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.448064+0000 front 2019-12-10T01:40:57.471258+0000 (oldest deadline 2019-12-10T01:41:37.879942+0000) 2019-12-10T01:41:42.369 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:41:42.359+0000 1694f700 -1 osd.7 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.445847+0000 front 2019-12-10T01:40:55.556959+0000 (oldest deadline 2019-12-10T01:41:39.401025+0000) 2019-12-10T01:41:42.520 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:41:42.510+0000 1694f700 -1 osd.6 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.546294+0000 front 2019-12-10T01:40:58.047498+0000 (oldest deadline 2019-12-10T01:41:39.762783+0000) 2019-12-10T01:41:42.925 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:41:42.914+0000 1694f700 -1 osd.5 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.396784+0000 front 2019-12-10T01:40:57.398288+0000 (oldest deadline 2019-12-10T01:41:39.196323+0000) 2019-12-10T01:41:43.125 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:41:43.120+0000 1694f700 -1 osd.2 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.046867+0000 front 2019-12-10T01:40:55.114340+0000 (oldest deadline 2019-12-10T01:41:40.884657+0000) 2019-12-10T01:41:43.251 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:43.247+0000 1694f700 -1 osd.0 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.470601+0000 front 2019-12-10T01:40:55.498885+0000 (oldest deadline 2019-12-10T01:41:40.154554+0000) 2019-12-10T01:41:43.300 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:41:43.294+0000 1694f700 -1 osd.4 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.448064+0000 front 2019-12-10T01:40:57.471258+0000 (oldest deadline 2019-12-10T01:41:37.879942+0000) 2019-12-10T01:41:43.414 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:41:43.408+0000 1694f700 -1 osd.7 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.445847+0000 front 2019-12-10T01:40:55.556959+0000 (oldest deadline 2019-12-10T01:41:39.401025+0000) 2019-12-10T01:41:43.529 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:41:43.523+0000 1694f700 -1 osd.6 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.546294+0000 front 2019-12-10T01:40:58.047498+0000 (oldest deadline 2019-12-10T01:41:39.762783+0000) 2019-12-10T01:41:43.891 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:41:43.877+0000 1694f700 -1 osd.5 157 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.396784+0000 front 2019-12-10T01:40:57.398288+0000 (oldest deadline 2019-12-10T01:41:39.196323+0000) 2019-12-10T01:41:44.195 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:41:44.157+0000 1694f700 -1 osd.2 158 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.046867+0000 front 2019-12-10T01:40:55.114340+0000 (oldest deadline 2019-12-10T01:41:40.884657+0000) 2019-12-10T01:41:44.259 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:44.252+0000 1694f700 -1 osd.0 158 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.470601+0000 front 2019-12-10T01:40:55.498885+0000 (oldest deadline 2019-12-10T01:41:40.154554+0000) 2019-12-10T01:41:44.335 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:44.307+0000 32f89700 -1 osd.0 158 build_incremental_map_msg missing incremental map 158 2019-12-10T01:41:44.346 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:44.311+0000 32f89700 -1 osd.0 158 build_incremental_map_msg missing incremental map 158 2019-12-10T01:41:44.351 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:41:44.330+0000 1694f700 -1 osd.4 158 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.448064+0000 front 2019-12-10T01:40:57.471258+0000 (oldest deadline 2019-12-10T01:41:37.879942+0000) 2019-12-10T01:41:44.431 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:44.421+0000 3378a700 -1 osd.0 158 build_incremental_map_msg missing incremental map 158 2019-12-10T01:41:44.457 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:41:44.433+0000 1694f700 -1 osd.7 158 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.445847+0000 front 2019-12-10T01:40:55.556959+0000 (oldest deadline 2019-12-10T01:41:39.401025+0000) 2019-12-10T01:41:44.503 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:44.447+0000 31f87700 -1 osd.0 158 build_incremental_map_msg missing incremental map 158 2019-12-10T01:41:44.565 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:41:44.553+0000 1694f700 -1 osd.6 158 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.546294+0000 front 2019-12-10T01:40:58.047498+0000 (oldest deadline 2019-12-10T01:41:39.762783+0000) 2019-12-10T01:41:44.597 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:44.575+0000 32f89700 -1 osd.0 158 build_incremental_map_msg missing incremental map 158 2019-12-10T01:41:44.872 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:41:44.864+0000 1694f700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.396784+0000 front 2019-12-10T01:40:57.398288+0000 (oldest deadline 2019-12-10T01:41:39.196323+0000) 2019-12-10T01:41:44.896 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:44.763+0000 36790700 -1 osd.0 158 build_incremental_map_msg missing incremental map 158 2019-12-10T01:41:45.125 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:41:45.120+0000 1694f700 -1 osd.2 159 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.046867+0000 front 2019-12-10T01:40:55.114340+0000 (oldest deadline 2019-12-10T01:41:40.884657+0000) 2019-12-10T01:41:45.329 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:41:45.313+0000 1694f700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:57.448064+0000 front 2019-12-10T01:40:57.471258+0000 (oldest deadline 2019-12-10T01:41:37.879942+0000) 2019-12-10T01:41:45.435 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:41:45.287+0000 1694f700 -1 osd.0 159 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:40:55.470601+0000 front 2019-12-10T01:40:55.498885+0000 (oldest deadline 2019-12-10T01:41:40.154554+0000) 2019-12-10T01:43:28.036 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:28.032+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:29.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:29.053+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:30.056 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:30.054+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:31.020 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:31.018+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:32.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:32.052+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:33.013 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:33.011+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:33.992 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:33.990+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:35.031 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:35.029+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:36.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:36.051+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:37.063 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:37.061+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:37.085 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:09:58.149 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:37.091 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:09:58.155 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:37.101 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:09:58.163 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:37.108 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:09:58.168 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:37.112 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:09:58.271 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:37.119 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:09:58.278 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:37.446 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:09:58.605 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:37.466 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:09:58.529 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:37.477 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:09:58.540 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:38.062 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:38.060+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:39.026 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:39.023+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:40.039 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:40.036+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:41.080 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:41.078+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:42.077 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:42.068+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:43.044 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:43.043+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:44.056 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:44.053+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:45.076 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:45.075+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:46.074 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:46.072+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:47.047 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:47.043+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:48.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:48.010+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:49.016 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:49.014+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:50.000 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:49.997+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:50.678 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:10:11.741 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:50.719 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:10:11.783 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:50.747 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:10:11.810 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:50.748 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:10:11.812 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:50.752 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:10:11.815 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:50.765 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:10:11.918 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:50.769 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:10:11.928 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:50.786 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:10:11.843 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:50.819 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:10:11.977 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:43:51.043 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:51.041+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:52.005 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:51.995+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:53.005 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:53.000+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:53.970 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:53.968+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:54.958 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:54.956+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:56.001 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:55.999+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:56.491 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:43:56.485+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:43:56.983 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:56.980+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:57.248 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:43:57.241+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:43:57.505 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:43:57.491+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:43:58.008 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:58.006+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:58.228 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:43:58.222+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:43:58.481 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:43:58.473+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:43:58.997 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:58.995+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:43:59.165 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:43:59.160+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:43:59.274 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:43:59.268+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:43:59.519 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:43:59.512+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:43:59.996 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:43:59.994+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:00.210 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:00.204+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:00.290 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:00.282+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:00.473 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:00.467+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:00.539 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:00.533+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:01.001 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:00.997+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:01.002 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:00.999+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:01.072 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:01.066+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:01.213 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:01.208+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:01.253 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:01.242+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:01.495 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:01.489+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:01.504 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:01.501+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:01.571 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:01.568+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:02.034 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:02.029+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:02.035 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:02.031+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:02.093 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:02.087+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:02.196 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:02.190+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:02.247 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:02.240+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:02.520 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:02.513+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:02.592 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:02.525+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:02.593 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:02.541+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:03.016 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:03.014+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:03.016 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:03.015+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:03.141 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:03.136+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:03.208 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:03.201+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:03.251 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:03.244+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:03.513 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:03.506+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:03.520 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:03.518+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:03.552 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:03.550+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:04.058 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:04.055+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:04.059 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:04.056+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:04.169 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:04.162+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:04.225 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:04.220+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:04.282 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:04.277+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:04.523 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:04.521+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:04.530 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:04.529+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:04.543 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:04.538+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:05.042 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:05.040+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:05.043 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:05.042+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:05.227 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:05.220+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:05.245 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:05.236+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:05.245 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:05.236+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:05.514 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:05.513+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:05.534 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:05.528+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:05.582 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:05.580+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:06.023 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:06.018+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:06.024 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:06.021+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:06.220 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:06.214+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:06.257 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:06.249+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:06.294 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:06.288+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:06.510 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:06.504+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:06.541 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:06.523+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:06.597 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:06.590+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:07.046 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:07.043+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:07.048 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:07.047+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:07.241 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:07.235+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:07.289 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:07.284+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:07.293 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:07.287+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:07.542 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:07.540+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:07.574 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:07.569+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:07.595 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:07.593+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:08.057 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:08.052+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:08.057 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:08.055+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:08.205 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:08.199+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:08.275 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:08.269+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:08.325 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:08.318+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:08.548 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:08.541+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:08.558 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:08.554+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:08.600 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:08.597+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:09.078 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:09.076+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:09.080 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:09.079+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:09.177 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:09.168+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:09.288 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:09.282+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:09.306 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:09.299+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:09.513 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:09.502+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:09.549 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:09.545+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:09.615 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:09.613+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:10.061 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:10.059+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:10.064 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:10.063+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:10.202 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:10.197+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:10.281 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:10.272+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:10.343 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:10.337+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:10.473 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:10.458+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:10.574 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:10.571+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:10.656 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:10.654+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:11.121 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:11.109+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:11.131 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:11.111+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:11.198 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:11.192+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:11.281 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:11.275+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:11.322 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:11.315+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:11.431 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:11.425+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:11.562 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:11.559+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:11.675 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:11.670+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:12.162 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:12.159+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:12.163 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:12.161+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:12.251 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:12.245+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:12.253 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:12.247+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:12.348 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:12.338+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:12.442 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:12.434+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:12.551 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:12.545+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:12.637 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:12.633+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:13.120 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:13.118+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:13.121 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:13.120+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:13.225 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:13.219+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:13.273 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:13.262+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:13.365 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:13.358+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:13.415 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:13.407+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:13.539 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:13.535+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:13.688 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:13.686+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:14.117 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:14.115+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:14.118 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:14.117+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:14.240 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:14.234+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:14.282 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:14.274+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:14.380 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:14.371+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:14.390 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:14.384+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:14.533 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:14.525+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:14.743 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:14.740+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:15.118 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:15.114+0000 1694f700 -1 osd.2 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.993798+0000 front 2019-12-10T01:43:14.992949+0000 (oldest deadline 2019-12-10T01:44:00.874406+0000) 2019-12-10T01:44:15.121 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:15.117+0000 1694f700 -1 osd.2 166 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:15.254 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:44:15.240+0000 1694f700 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.889337+0000 front 2019-12-10T01:43:15.801448+0000 (oldest deadline 2019-12-10T01:44:00.488202+0000) 2019-12-10T01:44:15.270 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:44:15.263+0000 1694f700 -1 osd.4 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:13.635484+0000 front 2019-12-10T01:43:13.895847+0000 (oldest deadline 2019-12-10T01:43:58.212336+0000) 2019-12-10T01:44:15.429 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:44:15.421+0000 1694f700 -1 osd.7 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:16.429032+0000 front 2019-12-10T01:43:11.719811+0000 (oldest deadline 2019-12-10T01:43:56.401429+0000) 2019-12-10T01:44:15.430 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:44:15.424+0000 1694f700 -1 osd.6 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:11.564951+0000 front 2019-12-10T01:43:11.406866+0000 (oldest deadline 2019-12-10T01:43:56.665053+0000) 2019-12-10T01:44:15.518 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:44:15.515+0000 1694f700 -1 osd.1 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:15.322830+0000 front 2019-12-10T01:43:15.324064+0000 (oldest deadline 2019-12-10T01:43:59.660783+0000) 2019-12-10T01:44:15.750 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:44:15.744+0000 1694f700 -1 osd.0 166 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:43:14.936120+0000 front 2019-12-10T01:43:14.946536+0000 (oldest deadline 2019-12-10T01:44:00.570970+0000) 2019-12-10T01:44:16.258 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:16.222+0000 1694f700 -1 osd.2 167 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:17.332 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:17.303+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:18.343 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:18.341+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:19.341 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:19.339+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:20.325 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:20.321+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:21.329 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:21.327+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:22.300 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:22.298+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:23.323 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:23.320+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:24.302 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:24.301+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:25.355 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:25.342+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:26.365 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:26.362+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:27.369 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:27.366+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:28.333 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:28.331+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:29.288 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:29.286+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:30.329 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:30.324+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:31.361 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:31.357+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:32.367 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:32.362+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:33.385 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:33.383+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:34.420 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:34.418+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:35.434 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:35.429+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:36.431 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:36.430+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:37.435 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:37.434+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:38.420 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:38.418+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:39.446 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:39.445+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:40.477 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:40.474+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:41.501 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:41.496+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:42.489 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:42.484+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:43.455 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:43.453+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:44.473 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:44.471+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:45.489 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:45.485+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:46.447 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:46.445+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:47.463 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:47.462+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:48.422 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:48.421+0000 1694f700 -1 osd.2 168 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:49.501 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:49.459+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:50.633 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:50.626+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:51.624 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:51.623+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:52.607 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:52.606+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:53.591 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:53.590+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:54.642 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:54.641+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:55.609 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:55.608+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:56.588 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:56.587+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:57.595 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:57.593+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:58.554 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:58.554+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:44:59.545 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:44:59.539+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:00.533 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:00.531+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:01.515 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:01.514+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:02.513 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:02.511+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:03.478 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:03.474+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:04.508 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:04.506+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:05.528 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:05.525+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:06.512 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:06.509+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:07.529 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:07.526+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:08.489 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:08.487+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:09.512 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:09.510+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:10.489 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:10.487+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:11.455 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:11.453+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:12.490 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:12.480+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:13.442 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:13.439+0000 1694f700 -1 osd.2 169 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:13.443 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:13.441+0000 1694f700 -1 osd.2 169 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~168 in=168b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e165) 2019-12-10T01:45:13.776 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:13.770+0000 1694f700 -1 osd.7 169 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:14.320 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:14.270+0000 1694f700 -1 osd.5 170 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:14.465 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:14.462+0000 1694f700 -1 osd.2 170 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:14.527 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:14.317+0000 1694f700 -1 osd.0 170 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:14.832 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:14.813+0000 1694f700 -1 osd.7 170 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:15.265 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:15.257+0000 1694f700 -1 osd.4 171 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:15.300 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:15.290+0000 1694f700 -1 osd.0 171 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:15.422 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:15.420+0000 1694f700 -1 osd.2 171 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:15.788 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:15.778+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:15.867 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:15.860+0000 1694f700 -1 osd.5 171 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:16.270 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:16.265+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:16.278 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:16.272+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:16.437 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:16.423+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:16.755 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:16.743+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:16.893 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:16.888+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:17.156 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:17.150+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:17.254 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:17.249+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:17.278 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:17.272+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:17.421 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:17.417+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:17.421 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:17.419+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:17.777 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:17.772+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:17.921 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:17.917+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:18.123 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:18.116+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:18.208 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:18.205+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:18.248 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:18.240+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:18.390 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:18.388+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:18.394 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:18.391+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:18.820 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:18.815+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:19.009 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:18.960+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:19.119 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:19.113+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:19.209 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:19.202+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:19.280 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:19.275+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:19.355 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:19.351+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:19.355 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:19.352+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:19.856 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:19.850+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:19.993 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:19.988+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:20.096 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:20.091+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:20.221 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:20.218+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:20.300 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:20.294+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:20.386 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:20.382+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:20.386 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:20.384+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:20.849 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:20.839+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:20.999 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:20.993+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:21.130 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:21.124+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:21.189 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:21.186+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:21.273 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:21.267+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:21.388 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:21.384+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:21.389 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:21.386+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:21.808 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:21.798+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:22.025 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:22.020+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:22.172 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:22.163+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:22.238 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:22.236+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:22.311 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:22.295+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:22.414 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:22.410+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:22.417 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:22.412+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:22.830 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:22.823+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:22.993 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:22.987+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:23.132 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:23.125+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:23.218 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:23.215+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:23.280 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:23.273+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:23.397 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:23.389+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:23.397 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:23.392+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:23.785 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:23.778+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:24.022 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:24.013+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:24.115 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:24.109+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:24.192 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:24.189+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:24.301 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:24.297+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:24.436 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:24.433+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:24.436 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:24.434+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:24.782 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:24.776+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:25.025 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:25.010+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:25.081 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:25.075+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:25.225 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:25.214+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:25.347 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:25.341+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:25.481 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:25.470+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:25.481 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:25.473+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:25.793 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:25.786+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:25.996 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:25.991+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:26.112 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:26.105+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:26.245 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:26.239+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:26.364 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:26.358+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:26.483 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:26.480+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:26.483 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:26.481+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:26.823 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:26.817+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:27.041 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:27.035+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:27.148 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:27.141+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:27.202 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:27.198+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:27.361 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:27.355+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:27.443 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:27.440+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:27.444 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:27.442+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:27.849 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:27.842+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:28.010 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:28.004+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:28.116 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:28.110+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:28.201 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:28.189+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:28.405 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:28.396+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:28.471 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:28.467+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:28.474 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:28.472+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:28.831 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:28.824+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:29.057 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:29.045+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:29.122 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:29.115+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:29.196 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:29.190+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:29.402 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:29.394+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:29.512 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:29.506+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:29.512 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:29.509+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:29.816 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:29.809+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:30.087 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:30.075+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:30.087 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:30.077+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:30.167 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:30.161+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:30.370 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:45:30.364+0000 1694f700 -1 osd.4 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:32.729231+0000 front 2019-12-10T01:44:32.757854+0000 (oldest deadline 2019-12-10T01:45:15.257286+0000) 2019-12-10T01:45:30.525 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:30.518+0000 1694f700 -1 osd.2 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.415509+0000 front 2019-12-10T01:44:31.597028+0000 (oldest deadline 2019-12-10T01:45:13.092308+0000) 2019-12-10T01:45:30.525 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:45:30.520+0000 1694f700 -1 osd.2 172 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:103940 2.3as0 2.edd193ba (undecoded) ondisk+retry+write+known_if_redirected e168) 2019-12-10T01:45:30.867 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:45:30.861+0000 1694f700 -1 osd.7 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.776321+0000 front 2019-12-10T01:44:31.882492+0000 (oldest deadline 2019-12-10T01:45:13.458603+0000) 2019-12-10T01:45:31.061 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:45:31.056+0000 1694f700 -1 osd.6 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.118599+0000 front 2019-12-10T01:44:31.099515+0000 (oldest deadline 2019-12-10T01:45:16.947208+0000) 2019-12-10T01:45:31.130 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:45:31.124+0000 1694f700 -1 osd.5 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:30.092515+0000 front 2019-12-10T01:44:30.164074+0000 (oldest deadline 2019-12-10T01:45:14.118778+0000) 2019-12-10T01:45:31.179 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:45:31.169+0000 1694f700 -1 osd.0 172 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:44:31.974022+0000 front 2019-12-10T01:44:31.940392+0000 (oldest deadline 2019-12-10T01:45:14.224589+0000) 2019-12-10T01:45:44.284 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:45:44.278+0000 1694f700 -1 osd.3 174 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104321 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5342034000 gen 6] snapc 0=[] ondisk+write+known_if_redirected e170) 2019-12-10T01:45:45.329 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:45:45.328+0000 1694f700 -1 osd.3 174 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104321 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5342034000 gen 6] snapc 0=[] ondisk+write+known_if_redirected e170) 2019-12-10T01:45:46.353 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:45:46.352+0000 1694f700 -1 osd.3 174 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104321 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5342034000 gen 6] snapc 0=[] ondisk+write+known_if_redirected e170) 2019-12-10T01:45:47.359 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:45:47.356+0000 1694f700 -1 osd.3 174 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4557.0:104321 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5342034000 gen 6] snapc 0=[] ondisk+write+known_if_redirected e170) 2019-12-10T01:45:48.416 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:45:48.409+0000 1694f700 -1 osd.3 174 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4557.0:104321 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5342034000 gen 6] snapc 0=[] ondisk+write+known_if_redirected e170) 2019-12-10T01:45:49.383 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:45:49.380+0000 1694f700 -1 osd.3 174 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4557.0:104321 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5342034000 gen 6] snapc 0=[] ondisk+write+known_if_redirected e170) 2019-12-10T01:45:50.434 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:45:50.431+0000 1694f700 -1 osd.3 174 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4557.0:104321 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5342034000 gen 6] snapc 0=[] ondisk+write+known_if_redirected e170) 2019-12-10T01:45:51.443 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:45:51.440+0000 1694f700 -1 osd.3 174 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4557.0:104321 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5342034000 gen 6] snapc 0=[] ondisk+write+known_if_redirected e170) 2019-12-10T01:45:52.424 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:45:52.422+0000 1694f700 -1 osd.3 174 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4557.0:104321 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 5342034000 gen 6] snapc 0=[] ondisk+write+known_if_redirected e170) 2019-12-10T01:46:38.122 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:46:38.077+0000 35f8f700 -1 osd.1 177 build_incremental_map_msg missing incremental map 177 2019-12-10T01:46:38.314 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:46:38.216+0000 32788700 -1 osd.1 177 build_incremental_map_msg missing incremental map 177 2019-12-10T01:46:38.488 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:46:38.224+0000 31f87700 -1 osd.1 177 build_incremental_map_msg missing incremental map 177 2019-12-10T01:47:13.580 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:13.579+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:14.610 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:14.609+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:15.578 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:15.576+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:16.622 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:16.620+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:17.581 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:17.579+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:18.608 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:18.606+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:19.563 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:19.561+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:20.594 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:20.592+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:21.631 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:21.630+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:22.621 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:22.618+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:23.589 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:23.587+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:24.610 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:24.608+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:24.725 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:13:45.788 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:47:24.728 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:13:45.792 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:47:24.750 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:13:45.909 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:47:24.759 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:13:45.918 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:47:24.794 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:13:45.857 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:47:24.797 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:13:45.861 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:47:25.007 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:13:46.070 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:47:25.050 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:13:46.209 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:47:25.119 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:13:46.182 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:47:25.663 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:25.660+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:26.615 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:26.613+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:27.607 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:27.601+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:28.566 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:28.565+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:29.566 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:29.564+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:30.594 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:30.592+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:31.622 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:31.620+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:32.611 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:32.609+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:33.603 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:33.601+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:34.560 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:34.558+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:35.606 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:35.604+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:36.568 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:36.566+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:37.573 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:37.570+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:37.577 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:37.574+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:37.791 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:37.783+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:38.558 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:38.553+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:38.559 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:38.556+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:38.819 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:38.810+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:39.436 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:39.429+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:39.552 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:39.549+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:39.601 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:39.597+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:39.602 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:39.600+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:39.759 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:39.753+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:39.799 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:39.794+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:40.100 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:40.093+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:40.482 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:40.476+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:40.580 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:40.577+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:40.604 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:40.601+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:40.605 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:40.603+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:40.735 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:40.730+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:40.768 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:40.763+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:41.106 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:41.100+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:41.400 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:41.397+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:41.530 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:41.520+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:41.542 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:41.537+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:41.602 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:41.599+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:41.603 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:41.601+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:41.784 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:41.776+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:41.805 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:41.799+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:42.124 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:42.118+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:42.416 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:42.413+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:42.526 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:42.521+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:42.548 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:42.543+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:42.616 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:42.613+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:42.616 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:42.614+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:42.813 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:42.807+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:42.814 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:42.809+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:43.111 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:43.105+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:43.463 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:43.459+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:43.520 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:43.513+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:43.560 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:43.556+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:43.606 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:43.603+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:43.607 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:43.606+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:43.778 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:43.768+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:43.812 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:43.805+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:44.089 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:44.080+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:44.464 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:44.461+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:44.546 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:44.540+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:44.595 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:44.591+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:44.657 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:44.653+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:44.658 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:44.655+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:44.736 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:44.730+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:44.813 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:44.807+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:45.055 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:45.049+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:45.511 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:45.508+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:45.542 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:45.536+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:45.605 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:45.598+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:45.620 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:45.617+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:45.620 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:45.618+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:45.745 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:45.739+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:45.794 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:45.783+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:46.106 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:46.100+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:46.518 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:46.512+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:46.539 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:46.535+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:46.593 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:46.589+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:46.637 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:46.633+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:46.639 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:46.636+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:46.789 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:46.783+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:46.824 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:46.818+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:47.137 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:47.124+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:47.526 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:47.518+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:47.557 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:47.553+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:47.600 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:47.596+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:47.621 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:47.618+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:47.622 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:47.621+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:47.759 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:47.753+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:47.804 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:47.798+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:48.163 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:48.156+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:48.521 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:48.519+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:48.557 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:48.554+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:48.579 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:48.573+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:48.610 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:48.606+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:48.611 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:48.608+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:48.741 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:48.735+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:48.843 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:48.837+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:49.159 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:49.153+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:49.521 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:49.511+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:49.521 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:49.517+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:49.537 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:49.529+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:49.625 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:49.622+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:49.626 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:49.624+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:49.775 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:49.769+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:49.859 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:49.854+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:50.177 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:50.171+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:50.514 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:50.511+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:50.549 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:50.542+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:50.576 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:50.573+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:50.665 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:50.661+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:50.665 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:50.664+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:50.818 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:50.812+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:50.881 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:50.876+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:51.228 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:51.222+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:51.527 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:51.521+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:51.561 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:51.559+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:51.569 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:51.567+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:51.680 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:51.676+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:51.681 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:51.679+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:51.784 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:51.779+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:51.934 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:51.923+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:52.195 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:52.190+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:52.566 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:52.560+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:52.566 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:52.552+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:52.576 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:52.567+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:52.662 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:52.661+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:52.663 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:52.662+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:52.761 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:52.755+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:52.926 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:52.920+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:53.160 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:53.155+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:53.561 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:53.558+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:53.592 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:53.588+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:53.597 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:53.591+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:53.709 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:53.706+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:53.712 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:53.711+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:53.769 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:53.762+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:53.934 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:53.928+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:54.129 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:54.123+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:54.577 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:54.576+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:54.606 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:54.600+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:54.640 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:54.638+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:54.736 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:54.734+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:54.737 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:54.736+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:54.764 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:54.759+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:54.956 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:54.950+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:55.122 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:55.111+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:55.603 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:55.598+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:55.624 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:55.621+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:55.686 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:55.684+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:55.725 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:55.723+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:55.726 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:55.725+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:55.751 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:55.745+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:55.966 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:55.959+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:56.117 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:56.112+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:56.606 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:56.599+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:56.662 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:56.660+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:56.689 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:56.686+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:56.742 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:56.737+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:56.745 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:56.738+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:56.792 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:56.786+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:56.979 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:56.973+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:57.166 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:57.159+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:57.583 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:57.574+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:57.700 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:57.698+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:57.741 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:57.731+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:57.783 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:57.777+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:57.785 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:57.782+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:57.786 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:57.785+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:57.958 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:57.953+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:58.191 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:58.183+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:58.573 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:58.567+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:58.661 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:58.660+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:58.720 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:58.719+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:58.759 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:58.756+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:58.759 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:58.757+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:58.806 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:58.795+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:47:58.978 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:47:58.971+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:47:59.152 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:47:59.147+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:47:59.609 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:47:59.600+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:47:59.688 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:47:59.686+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:47:59.756 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:47:59.753+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:47:59.771 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:59.767+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:47:59.773 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:47:59.769+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:47:59.822 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:47:59.808+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:00.011 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:00.004+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:00.141 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:00.131+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:00.619 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:00.612+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:00.716 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:00.713+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:00.734 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:00.731+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:00.734 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:00.733+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:00.744 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:00.742+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:00.793 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:00.787+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:01.004 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:00.997+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:01.133 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:01.126+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:01.610 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:01.604+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:01.693 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:01.690+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:01.693 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:01.691+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:01.694 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:01.689+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:01.755 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:01.743+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:01.793 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:01.782+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:02.034 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:02.028+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:02.103 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:02.097+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:02.660 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:02.654+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:02.677 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:02.672+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:02.679 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:02.675+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:02.722 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:02.719+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:02.753 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:02.749+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:02.757 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:02.752+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:03.064 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:03.058+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:03.114 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:03.106+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:03.648 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:03.638+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:03.648 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:03.642+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:03.656 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:03.649+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:03.719 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:03.714+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:03.792 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:03.786+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:03.798 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:03.791+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:04.106 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:04.101+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:04.138 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:04.132+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:04.667 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:04.665+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:04.669 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:04.667+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:04.688 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:04.682+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:04.704 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:04.701+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:04.751 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:04.748+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:04.846 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:04.840+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:05.067 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:05.062+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:05.168 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:05.161+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:05.689 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:05.681+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:05.689 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:05.682+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:05.709 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:05.701+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:05.709 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:05.700+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:05.713 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:05.710+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:05.835 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:05.830+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:06.114 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:06.106+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:06.175 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:06.168+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:06.650 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:06.646+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:06.650 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:06.648+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:06.663 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:06.659+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:06.701 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:06.697+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:06.750 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:06.744+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:06.794 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:06.786+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:07.141 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:07.134+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:07.207 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:07.196+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:07.643 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:07.635+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:07.644 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:07.638+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:07.645 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:07.639+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:07.667 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:07.664+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:07.718 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:07.712+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:07.797 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:07.790+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:08.118 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:08.110+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:08.243 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:08.237+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:08.608 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:08.600+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:08.608 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:08.602+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:08.681 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:08.677+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:08.712 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:08.707+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:08.730 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:08.723+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:08.754 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:08.747+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:09.130 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:09.122+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:09.248 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:09.242+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:09.603 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:09.599+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:09.604 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:09.601+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:09.675 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:09.670+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:09.710 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:09.704+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:09.724 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:09.720+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:09.732 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:09.726+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:10.160 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:10.152+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:10.224 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:10.216+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:10.565 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:10.562+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:10.565 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:10.563+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:10.705 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:10.696+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:10.714 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:10.709+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:10.715 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:10.709+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:10.729 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:10.724+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:11.188 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:48:11.177+0000 1694f700 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.546838+0000 front 2019-12-10T01:46:57.573447+0000 (oldest deadline 2019-12-10T01:47:39.691335+0000) 2019-12-10T01:48:11.213 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:48:11.206+0000 1694f700 -1 osd.7 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:58.019957+0000 front 2019-12-10T01:46:58.032490+0000 (oldest deadline 2019-12-10T01:47:39.684874+0000) 2019-12-10T01:48:11.597 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:11.591+0000 1694f700 -1 osd.2 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:52.852429+0000 front 2019-12-10T01:46:57.576464+0000 (oldest deadline 2019-12-10T01:47:37.546438+0000) 2019-12-10T01:48:11.597 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:11.593+0000 1694f700 -1 osd.2 178 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:11.724 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:48:11.715+0000 1694f700 -1 osd.4 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:56.272322+0000 front 2019-12-10T01:46:56.267108+0000 (oldest deadline 2019-12-10T01:47:38.874925+0000) 2019-12-10T01:48:11.726 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:11.723+0000 1694f700 -1 osd.0 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:55.866145+0000 front 2019-12-10T01:46:55.842533+0000 (oldest deadline 2019-12-10T01:47:38.673514+0000) 2019-12-10T01:48:11.734 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:48:11.728+0000 1694f700 -1 osd.6 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:57.801804+0000 front 2019-12-10T01:46:53.310286+0000 (oldest deadline 2019-12-10T01:47:37.657552+0000) 2019-12-10T01:48:11.737 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:48:11.725+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 45sec e178 v178) 2019-12-10T01:48:11.753 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:48:11.750+0000 1694f700 -1 osd.1 178 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:46:54.521754+0000 front 2019-12-10T01:46:54.528571+0000 (oldest deadline 2019-12-10T01:47:40.408283+0000) 2019-12-10T01:48:12.604 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:12.600+0000 1694f700 -1 osd.2 179 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:13.669 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:13.667+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:14.689 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:14.685+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:15.714 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:15.713+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:16.706 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:16.704+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:17.757 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:17.755+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:18.773 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:18.771+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:19.767 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:19.765+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:20.820 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:20.815+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:21.788 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:21.785+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:22.829 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:22.821+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:23.833 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:23.831+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:24.812 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:24.809+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:25.772 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:25.769+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:26.781 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:26.776+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:27.818 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:27.816+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:28.849 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:28.841+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:29.885 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:29.884+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:30.879 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:30.877+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:31.839 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:31.837+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:32.872 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:32.868+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:33.882 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:33.880+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:34.907 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:34.906+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:35.897 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:35.888+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:36.864 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:36.863+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:37.889 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:37.888+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:38.884 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:38.883+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:39.876 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:39.875+0000 1694f700 -1 osd.2 180 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:40.849 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:40.842+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:41.857 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:41.855+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:42.824 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:42.823+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:43.812 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:43.811+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:44.851 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:44.851+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:45.804 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:45.803+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:46.802 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:46.798+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:47.850 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:47.848+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:48.870 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:48.868+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:49.907 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:49.905+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:50.879 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:50.877+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:51.893 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:51.891+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:52.849 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:52.847+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:53.820 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:53.818+0000 1694f700 -1 osd.2 181 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:54.098 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:54.011+0000 36f91700 -1 osd.0 182 build_incremental_map_msg missing incremental map 182 2019-12-10T01:48:54.154 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:54.077+0000 36790700 -1 osd.0 182 build_incremental_map_msg missing incremental map 182 2019-12-10T01:48:54.160 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:54.129+0000 31f87700 -1 osd.0 182 build_incremental_map_msg missing incremental map 182 2019-12-10T01:48:54.185 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:54.179+0000 32788700 -1 osd.0 182 build_incremental_map_msg missing incremental map 182 2019-12-10T01:48:54.725 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:48:54.606+0000 3378a700 -1 osd.0 182 build_incremental_map_msg missing incremental map 182 2019-12-10T01:48:54.986 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:54.984+0000 1694f700 -1 osd.2 183 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:55.964 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:55.962+0000 1694f700 -1 osd.2 183 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:57.020 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:57.017+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:58.050 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:58.040+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:48:59.046 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:48:59.044+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:00.084 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:00.082+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:01.101 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:01.096+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:02.119 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:02.115+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:03.082 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:03.080+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:04.126 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:04.117+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:05.079 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:05.078+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:06.088 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:06.085+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:07.082 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:07.080+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:08.086 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:08.084+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:09.109 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:09.107+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:10.062 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:10.060+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:10.120 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:10.114+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:10.178 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:10.172+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:10.597 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:10.591+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:10.612 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:10.608+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:11.109 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:11.108+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:11.110 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:11.105+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:11.154 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:11.148+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:11.644 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:11.635+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:11.658 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:11.655+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:12.117 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:12.108+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:12.161 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:12.145+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:12.161 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:12.152+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:12.291 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:12.285+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:12.625 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:12.620+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:12.626 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:12.618+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:13.086 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:13.083+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:13.086 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:13.084+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:13.120 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:13.109+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:13.136 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:13.120+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:13.329 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:13.324+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:13.584 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:13.580+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:13.621 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:13.614+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:14.048 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:14.045+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:14.050 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:14.048+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:14.074 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:14.066+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:14.177 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:14.167+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:14.325 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:14.319+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:14.638 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:14.632+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:14.669 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:14.663+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:15.040 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:15.032+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:15.041 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:15.033+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:15.084 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:15.078+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:15.223 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:15.216+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:15.283 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:15.276+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:15.649 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:15.640+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:15.649 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:15.638+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:16.029 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:16.023+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:16.029 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:16.026+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:16.059 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:16.053+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:16.220 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:16.211+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:16.252 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:16.245+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:16.601 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:16.595+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:16.611 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:16.605+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:17.020 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:17.012+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:17.063 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:17.055+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:17.064 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:17.057+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:17.179 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:17.171+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:17.232 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:17.225+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:17.567 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:17.564+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:17.593 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:17.587+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:17.993 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:17.985+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:18.093 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:18.088+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:18.093 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:18.091+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:18.167 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:18.161+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:18.204 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:18.198+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:18.576 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:18.575+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:18.628 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:18.622+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:19.007 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:18.996+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:19.125 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:19.119+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:19.133 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:19.131+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:19.134 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:19.133+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:19.232 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:19.225+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:19.589 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:19.584+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:19.671 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:19.665+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:20.010 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:20.004+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:20.160 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:20.152+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:20.168 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:20.165+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:20.169 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:20.168+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:20.200 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:20.194+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:20.584 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:20.582+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:20.649 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:20.643+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:20.992 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:20.986+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:21.128 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:21.126+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:21.129 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:21.128+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:21.168 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:21.162+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:21.241 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:21.235+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:21.617 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:21.609+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:21.645 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:21.639+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:21.960 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:21.952+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:22.082 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:22.081+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:22.083 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:22.082+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:22.207 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:22.200+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:22.216 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:22.210+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:22.605 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:22.598+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:22.641 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:22.638+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:22.933 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:22.919+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:23.091 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:23.090+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:23.092 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:23.091+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:23.188 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:23.182+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:23.207 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:23.197+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:23.616 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:23.611+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:23.631 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:23.628+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:23.946 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:23.940+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:24.134 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:24.132+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:24.135 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:24.133+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:24.141 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:24.136+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:24.237 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:24.232+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:24.467 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:24.466+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:24.586 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:24.583+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:24.665 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:24.656+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:24.954 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:24.948+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:25.129 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:25.127+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:25.131 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:25.130+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:25.174 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:25.167+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:25.264 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:25.256+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:25.467 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:25.466+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:25.608 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:25.605+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:25.651 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:25.644+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:25.918 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:25.913+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:26.152 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:26.144+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:26.177 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:26.173+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:26.178 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:26.176+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:26.298 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:26.291+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:26.502 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:26.499+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:26.621 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:26.612+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:26.641 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:26.635+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:26.962 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:26.956+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:27.164 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:27.159+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:27.213 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:27.210+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:27.213 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:27.211+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:27.337 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:27.330+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:27.492 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:27.491+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:27.632 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:27.628+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:27.647 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:27.640+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:27.956 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:27.950+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:28.152 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:28.146+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:28.222 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:28.220+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:28.223 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:28.222+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:28.353 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:28.347+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:28.451 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:28.450+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:28.618 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:28.616+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:28.667 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:28.660+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:28.960 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:28.953+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:29.154 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:29.147+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:29.241 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:29.238+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:29.241 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:29.240+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:29.372 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:29.365+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:29.511 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:29.492+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:29.514 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:29.508+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:29.673 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:29.671+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:29.713 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:29.706+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:29.950 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:29.943+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:30.119 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:30.113+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:30.226 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:30.224+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:30.227 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:30.226+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:30.388 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:30.373+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:30.521 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:30.518+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:30.523 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:30.521+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:30.641 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:30.636+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:30.751 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:30.744+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:30.913 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:30.906+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:31.093 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:31.082+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:31.215 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:31.213+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:31.237 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:31.214+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:31.416 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:31.404+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:31.505 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:31.502+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:31.506 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:31.504+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:31.594 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:31.592+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:31.747 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:31.741+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:31.969 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:31.960+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:32.140 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:32.134+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:32.218 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:32.213+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:32.218 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:32.216+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:32.375 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:15:53.432 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:49:32.376 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:15:53.439 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:49:32.433 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:32.427+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:32.435 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:15:53.498 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:49:32.438 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:15:53.502 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:49:32.465 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:32.460+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:32.465 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:32.462+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:32.483 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:15:53.642 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:49:32.491 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:15:53.650 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:49:32.600 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:15:53.661 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:49:32.609 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:15:53.673 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:49:32.628 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:32.623+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:32.718 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:32.712+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:32.827 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:15:53.985 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:49:32.970 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:32.963+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:33.172 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:33.166+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:33.201 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:33.198+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:33.203 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:33.201+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:33.441 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:33.433+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:33.441 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:33.436+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:33.484 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:33.477+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:33.609 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:33.602+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:33.689 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:33.679+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:33.935 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:33.929+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:34.182 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:34.179+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:34.182 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:34.180+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:34.191 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:34.186+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:34.490 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:34.486+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:34.490 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:34.488+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:34.513 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:34.504+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:34.589 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:34.585+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:34.677 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:34.669+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:34.963 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:34.957+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:35.160 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:35.153+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:35.160 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:35.155+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:35.249 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:35.225+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:35.530 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:35.526+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:35.532 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:35.529+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:35.549 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:35.540+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:35.565 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:35.559+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:35.666 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:35.660+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:35.938 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:35.931+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:36.122 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:36.117+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:36.122 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:36.119+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:36.255 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:36.248+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:36.496 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:36.492+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:36.499 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:36.497+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:36.528 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:36.522+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:36.534 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:36.527+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:36.720 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:36.713+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:36.942 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:36.935+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:37.147 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:37.144+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:37.148 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:37.146+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:37.226 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:37.221+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:37.491 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:37.487+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:37.493 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:37.491+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:37.555 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:37.543+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:37.570 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:37.563+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:37.709 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:37.702+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:37.954 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:37.945+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:38.103 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:38.100+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:38.107 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:38.101+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:38.258 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:38.251+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:38.506 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:38.502+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:38.508 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:38.506+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:38.524 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:38.521+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:38.537 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:38.528+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:38.741 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:38.735+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:38.915 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:38.908+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:39.151 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:39.147+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:39.151 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:39.148+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:39.305 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:39.297+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:39.509 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:39.499+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:39.509 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:39.501+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:39.533 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:39.529+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:39.580 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:39.573+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:39.788 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:39.781+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:39.953 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:39.941+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:40.108 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:40.104+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:40.110 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:40.108+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:40.324 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:40.316+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:40.472 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:40.468+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:40.474 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:40.472+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:40.507 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:40.503+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:40.590 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:40.583+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:40.789 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:40.783+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:40.949 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:40.942+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:41.149 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:41.145+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:41.149 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:41.147+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:41.297 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:41.271+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:41.532 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:41.524+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:41.532 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:41.526+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:41.549 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:41.545+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:41.604 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:41.598+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:41.769 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:41.762+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:41.949 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:41.941+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:42.139 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:42.136+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:42.140 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:42.137+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:42.292 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:42.282+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:42.304 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:49:42.299+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e184 v184) 2019-12-10T01:49:42.515 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:42.513+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:42.518 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:42.516+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:42.577 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:42.575+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:42.628 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:42.622+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:42.809 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:42.785+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:42.944 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:42.938+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:43.127 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:43.122+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:43.127 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:43.123+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:43.284 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:43.278+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:43.487 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:43.480+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:43.488 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:43.484+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:43.629 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:43.624+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:43.665 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:43.655+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:43.814 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:43.787+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:43.919 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:43.912+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:44.094 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:44.091+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:44.095 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:44.093+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:44.320 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:44.314+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:44.446 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:44.443+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:44.447 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:44.444+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:44.636 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:44.631+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:44.718 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:44.703+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:44.758 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:44.750+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:44.957 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:44.950+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:45.125 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:45.122+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:45.126 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:45.124+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:45.323 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:45.317+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:45.458 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:45.455+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:45.459 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:45.457+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:45.622 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:45.618+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:45.730 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:45.723+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:45.764 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:45.756+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:46.012 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:46.002+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:46.124 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:46.119+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:46.124 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:46.121+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:46.324 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:46.318+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:46.452 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:46.447+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:46.452 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:46.450+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:46.642 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:46.639+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:46.779 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:46.774+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:46.783 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:46.777+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:47.053 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:47.046+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:47.083 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:47.080+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:47.083 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:47.081+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:47.292 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:47.285+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:47.321 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:49:47.316+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e184 v184) 2019-12-10T01:49:47.420 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:47.417+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:47.423 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:47.421+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:47.693 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:47.688+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:47.743 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:47.736+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:47.821 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:47.814+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:48.068 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:48.060+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:48.083 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:48.079+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:48.086 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:48.081+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:48.259 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:48.251+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:48.428 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:48.426+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:48.430 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:48.428+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:48.660 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:48.657+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:48.724 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:48.717+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:48.844 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:48.838+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:49.094 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:49.088+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:49.120 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:49.117+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:49.121 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:49.118+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:49.263 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:49.257+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:49.390 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:49.386+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:49.391 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:49.389+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:49.628 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:49.623+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:49.695 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:49.689+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:49.813 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:49.807+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:50.080 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:50.068+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:50.103 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:50.099+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:50.104 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:50.101+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:50.257 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:50.250+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:50.424 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:50.421+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:50.426 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:50.424+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:50.615 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:50.611+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:50.663 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:50.653+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:50.802 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:50.798+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:51.083 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:51.078+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:51.157 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:51.148+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:51.157 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:51.149+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:51.231 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:51.219+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:51.398 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:51.394+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:51.398 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:51.396+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:51.620 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:51.613+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:51.646 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:51.642+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:51.837 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:51.832+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:52.115 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:52.108+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:52.120 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:52.117+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:52.122 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:52.121+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:52.200 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:52.194+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:52.345 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:49:52.340+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e184 v184) 2019-12-10T01:49:52.408 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:52.401+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:52.408 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:52.403+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:52.665 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:52.657+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:52.667 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:52.662+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:52.887 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:52.880+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:53.071 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:53.065+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:53.103 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:53.099+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:53.105 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:53.103+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:53.170 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:53.162+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:53.435 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:53.431+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:53.438 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:53.436+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:53.680 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:49:53.675+0000 1694f700 -1 osd.7 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.774664+0000 front 2019-12-10T01:48:28.780593+0000 (oldest deadline 2019-12-10T01:49:09.252345+0000) 2019-12-10T01:49:53.719 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:53.716+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:53.855 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:53.848+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:54.083 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:54.076+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:54.096 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:54.090+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:54.096 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:54.093+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:54.181 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:54.175+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:54.445 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:54.440+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:54.446 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:54.444+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:54.693 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:54.690+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:54.883 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:54.877+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:55.077 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:55.071+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:55.104 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:55.101+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:55.105 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:55.103+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:55.181 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:55.175+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:55.419 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:55.414+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:55.419 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:55.417+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:55.691 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:55.684+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:55.892 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:55.886+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:56.039 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:56.033+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:56.139 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:56.135+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:56.142 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:56.140+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:56.187 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:56.182+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:56.460 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:56.456+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:56.460 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:56.458+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:56.704 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:56.700+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:56.948 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:56.941+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:57.082 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:57.076+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:57.146 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:57.141+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:57.146 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:57.143+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:57.168 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:57.162+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:57.363 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:49:57.360+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e184 v184) 2019-12-10T01:49:57.433 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:57.414+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:57.433 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:57.418+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:57.683 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:57.679+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:57.978 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:57.966+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:58.045 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:58.038+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:58.138 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:58.132+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:58.171 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:58.166+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:58.171 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:58.168+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:58.398 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:58.394+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:58.399 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:58.396+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:58.685 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:58.680+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:49:58.968 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:49:58.962+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:49:59.061 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:49:59.053+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:49:59.176 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:49:59.170+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:49:59.186 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:59.183+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:49:59.187 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:49:59.185+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:49:59.378 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:59.375+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:49:59.380 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:49:59.378+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:49:59.704 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:49:59.701+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:50:00.021 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:50:00.016+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:50:00.114 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:50:00.108+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:50:00.197 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:50:00.194+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:50:00.198 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:50:00.197+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:50:00.217 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:50:00.211+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:50:00.401 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:50:00.397+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:50:00.401 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:50:00.399+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:50:00.727 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:50:00.723+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:50:01.066 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:50:01.060+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:50:01.110 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:50:01.102+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:50:01.172 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:50:01.168+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:50:01.172 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:50:01.169+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:50:01.231 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:50:01.225+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:50:01.380 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:50:01.376+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:50:01.382 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:50:01.380+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:50:01.702 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:50:01.699+0000 1694f700 -1 osd.0 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:25.590764+0000 front 2019-12-10T01:48:25.658141+0000 (oldest deadline 2019-12-10T01:49:10.250158+0000) 2019-12-10T01:50:02.049 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:50:02.042+0000 1694f700 -1 osd.6 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.966319+0000 front 2019-12-10T01:48:27.916059+0000 (oldest deadline 2019-12-10T01:49:10.169640+0000) 2019-12-10T01:50:02.090 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:50:02.084+0000 1694f700 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:24.538052+0000 front 2019-12-10T01:48:24.579836+0000 (oldest deadline 2019-12-10T01:49:09.808901+0000) 2019-12-10T01:50:02.197 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:50:02.195+0000 1694f700 -1 osd.2 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:27.428305+0000 front 2019-12-10T01:48:27.557246+0000 (oldest deadline 2019-12-10T01:49:12.653951+0000) 2019-12-10T01:50:02.200 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:50:02.198+0000 1694f700 -1 osd.2 184 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] ondisk+read+known_if_redirected e178) 2019-12-10T01:50:02.240 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:50:02.235+0000 1694f700 -1 osd.4 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:48:28.613949+0000 front 2019-12-10T01:48:28.622858+0000 (oldest deadline 2019-12-10T01:49:11.981424+0000) 2019-12-10T01:50:02.415 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:50:02.412+0000 1694f700 -1 osd.3 184 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:48:49.343181+0000 (oldest deadline 2019-12-10T01:49:29.343181+0000) 2019-12-10T01:50:02.417 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:50:02.415+0000 1694f700 -1 osd.3 184 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e182) 2019-12-10T01:50:02.444 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:50:02.433+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e184 v184) 2019-12-10T01:50:57.330 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:17:18.393 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:50:57.341 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:17:18.404 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:50:57.346 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:17:18.410 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:50:57.349 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:17:18.413 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:50:57.377 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:17:18.522 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:50:57.377 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:17:18.533 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:50:57.490 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:17:18.554 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:50:57.503 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:17:18.566 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:50:57.550 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:17:18.708 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:51:03.478 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:03.476+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:04.519 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:04.517+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:05.561 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:05.555+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:06.564 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:06.561+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:07.582 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:07.580+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:08.628 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:08.625+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:09.613 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:09.610+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:10.636 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:10.634+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:11.648 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:11.645+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:12.689 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:12.686+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:13.682 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:13.680+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:14.713 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:14.711+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:15.717 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:15.716+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:16.675 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:16.673+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:17.668 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:17.665+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:18.623 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:18.621+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:19.610 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:19.608+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:20.567 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:20.565+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:21.610 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:21.608+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:22.591 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:22.586+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:23.610 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:23.607+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:24.645 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:24.642+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:25.630 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:25.628+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:26.655 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:26.653+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:27.668 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:27.666+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:28.700 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:28.697+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:29.736 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:29.734+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:30.772 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:30.770+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:31.788 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:31.786+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:32.826 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:32.825+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:33.751 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:33.743+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:33.837 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:33.836+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:33.931 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:33.928+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:34.213 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:34.206+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:34.754 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:34.748+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:34.780 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:34.773+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:34.830 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:34.827+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:34.831 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:34.829+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:34.887 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:34.883+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:34.889 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:34.888+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:35.190 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:35.184+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:35.718 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:35.712+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:35.762 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:35.755+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:35.784 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:35.781+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:35.785 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:35.783+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:35.857 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:35.855+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:35.904 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:35.902+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:36.204 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:36.199+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:36.684 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:36.678+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:36.757 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:36.751+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:36.795 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:36.793+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:36.796 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:36.795+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:36.850 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:36.848+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:36.915 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:36.913+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:37.111 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:37.105+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:37.248 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:37.241+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:37.697 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:37.692+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:37.773 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:37.768+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:37.791 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:37.788+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:37.792 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:37.790+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:37.853 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:37.850+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:37.960 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:37.959+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:38.148 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:38.138+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:38.269 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:38.262+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:38.686 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:38.680+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:38.743 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:38.737+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:38.823 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:38.821+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:38.825 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:38.824+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:38.861 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:38.857+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:38.969 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:38.966+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:39.160 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:39.154+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:39.322 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:39.316+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:39.657 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:39.652+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:39.785 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:39.779+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:39.837 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:39.831+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:39.877 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:39.872+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:39.885 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:39.873+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:39.964 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:39.958+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:40.185 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:40.171+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:40.291 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:40.286+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:40.689 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:40.682+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:40.742 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:40.736+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:40.844 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:40.842+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:40.899 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:40.897+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:40.903 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:40.901+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:41.015 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:41.005+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:41.140 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:41.134+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:41.260 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:41.254+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:41.696 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:41.690+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:41.760 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:41.754+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:41.867 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:41.864+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:41.950 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:41.947+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:41.950 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:41.949+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:41.993 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:41.990+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:42.176 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:42.170+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:42.240 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:42.231+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:42.683 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:42.677+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:42.752 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:42.746+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:42.908 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:42.906+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:42.960 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:42.956+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:42.960 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:42.959+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:43.033 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:43.030+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:43.229 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:43.222+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:43.257 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:43.251+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:43.722 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:43.716+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:43.804 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:43.796+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:43.890 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:43.887+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:43.958 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:43.956+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:43.960 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:43.959+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:43.990 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:43.988+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:44.269 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:44.263+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:44.277 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:44.268+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:44.756 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:44.750+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:44.790 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:44.783+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:44.916 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:44.913+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:44.944 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:44.942+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:44.945 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:44.943+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:45.021 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:45.018+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:45.303 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:45.297+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:45.317 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:45.311+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:45.797 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:45.790+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:45.826 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:45.820+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:45.906 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:45.903+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:45.982 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:45.980+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:45.995 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:45.993+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:45.995 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:45.994+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:46.293 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:46.283+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:46.340 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:46.334+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:46.790 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:46.784+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:46.846 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:46.841+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:46.889 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:46.885+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:46.988 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:46.984+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:47.004 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:47.001+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:47.005 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:47.002+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:47.250 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:47.245+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:47.352 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:47.347+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:47.757 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:47.751+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:47.839 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:47.833+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:47.843 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:47.840+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:47.948 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:47.945+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:47.988 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:47.985+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:47.991 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:47.989+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:48.218 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:48.213+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:48.353 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:48.348+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:48.733 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:48.725+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:48.855 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:48.850+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:48.866 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:48.860+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:48.914 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:48.911+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:48.977 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:48.973+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:48.978 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:48.974+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:49.222 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:49.215+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:49.361 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:49.352+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:49.772 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:49.767+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:49.872 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:49.868+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:49.903 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:49.896+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:49.943 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:49.937+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:50.005 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:50.003+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:50.006 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:50.004+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:50.252 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:50.244+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:50.356 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:50.351+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:50.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:50.801+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:50.881 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:50.878+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:50.930 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:50.924+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:50.967 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:50.963+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:50.968 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:50.965+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:50.993 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:50.989+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:51.283 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:51.275+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:51.408 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:51.402+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:51.817 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:51.810+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:51.878 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:51.875+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:51.947 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:51.944+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:51.949 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:51.947+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:51.961 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:51.952+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:52.002 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:51.999+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:52.321 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:52.315+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:52.373 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:52.364+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:52.781 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:52.775+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:52.860 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:52.855+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:52.991 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:52.988+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:52.992 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:52.985+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:52.994 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:52.992+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:53.001 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:52.997+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:53.307 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:53.292+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:53.400 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:53.393+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:53.831 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:53.825+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:53.869 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:53.864+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:53.948 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:53.945+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:53.948 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:53.946+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:53.972 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:53.970+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:54.008 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:54.003+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:54.334 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:54.327+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:54.368 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:54.360+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:54.820 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:54.815+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:54.899 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:54.895+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:54.933 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:54.930+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:54.985 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:54.982+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:54.987 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:54.985+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:55.037 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:55.032+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:55.320 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:55.315+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:55.327 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:55.321+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:55.781 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:55.776+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:55.891 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:55.888+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:55.912 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:55.907+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:55.947 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:55.944+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:55.948 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:55.946+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:56.071 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:56.066+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:56.322 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:56.316+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:56.336 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:56.324+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:56.779 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:56.773+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:56.870 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:56.866+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:56.900 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:56.896+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:56.955 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:56.950+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:56.955 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:56.952+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:57.056 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:57.050+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:57.346 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:57.339+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:57.371 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:57.366+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:57.820 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:57.814+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:57.859 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:57.855+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:57.942 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:57.939+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:57.945 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:57.942+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:57.946 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:57.944+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:58.033 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:58.027+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:58.302 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:58.297+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:58.389 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:58.383+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:58.831 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:58.825+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:58.858 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:58.854+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:58.910 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:58.906+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:58.911 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:58.908+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:51:58.980 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:51:58.976+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:51:59.072 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:51:59.064+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:51:59.275 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:51:59.268+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:51:59.420 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:51:59.414+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:51:59.820 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:51:59.814+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:51:59.845 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:51:59.838+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:51:59.926 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:59.922+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:51:59.928 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:51:59.926+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:00.028 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:52:00.025+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:52:00.031 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:52:00.025+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:52:00.256 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:52:00.247+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:52:00.421 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:52:00.415+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:52:00.810 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:52:00.804+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:52:00.885 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:00.881+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:52:00.913 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:00.910+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:52:00.916 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:00.913+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:01.018 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:52:01.014+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:52:01.056 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:52:01.048+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:52:01.279 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:52:01.270+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:52:01.455 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:52:01.449+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:52:01.839 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:52:01.830+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:52:01.875 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:01.865+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:52:01.965 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:01.960+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:52:01.966 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:01.963+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:02.069 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:52:02.062+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:52:02.073 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:52:02.068+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:52:02.233 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:52:02.227+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:52:02.512 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:52:02.501+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:52:02.823 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:52:02.816+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:52:02.870 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:02.866+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:52:02.986 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:02.983+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:52:02.986 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:02.984+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:03.030 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:52:03.025+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:52:03.108 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:52:03.102+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:52:03.281 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:52:03.275+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:52:03.510 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:52:03.505+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:52:03.831 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:52:03.825+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:52:03.833 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:03.830+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:52:04.010 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:04.007+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:52:04.011 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:04.008+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:04.042 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:52:04.038+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:52:04.108 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:52:04.103+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:52:04.287 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:52:04.280+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:52:04.525 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:52:04.519+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:52:04.841 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:04.834+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:52:04.847 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:52:04.841+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:52:05.038 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:05.020+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:52:05.039 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:05.023+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:05.093 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:52:05.085+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:52:05.100 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:52:05.088+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:52:05.303 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:52:05.297+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:52:05.488 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:52:05.482+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:52:05.817 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:05.815+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:52:05.843 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:52:05.838+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:52:05.980 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:05.976+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:52:05.980 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:05.977+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:06.054 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:52:06.049+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:52:06.087 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:52:06.084+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:52:06.278 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:52:06.271+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:52:06.521 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:52:06.515+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:52:06.853 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:06.848+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:52:06.891 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:52:06.885+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:52:06.954 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:06.949+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:52:06.955 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:06.952+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:07.031 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:52:07.023+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:52:07.075 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:52:07.071+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:52:07.321 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:52:07.315+0000 1694f700 -1 osd.7 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.294444+0000 front 2019-12-10T01:50:51.296788+0000 (oldest deadline 2019-12-10T01:51:37.068114+0000) 2019-12-10T01:52:07.480 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:52:07.473+0000 1694f700 -1 osd.6 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:48.461274+0000 front 2019-12-10T01:50:48.454490+0000 (oldest deadline 2019-12-10T01:51:34.049292+0000) 2019-12-10T01:52:07.875 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:07.872+0000 1694f700 -1 osd.0 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.221199+0000 front 2019-12-10T01:50:51.279062+0000 (oldest deadline 2019-12-10T01:51:34.557344+0000) 2019-12-10T01:52:07.877 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:52:07.870+0000 1694f700 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:50.918603+0000 front 2019-12-10T01:50:50.916530+0000 (oldest deadline 2019-12-10T01:51:34.365827+0000) 2019-12-10T01:52:07.915 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:07.911+0000 1694f700 -1 osd.2 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:52.374814+0000 front 2019-12-10T01:50:52.387200+0000 (oldest deadline 2019-12-10T01:51:34.060562+0000) 2019-12-10T01:52:07.915 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:07.913+0000 1694f700 -1 osd.2 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:08.073 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:52:08.067+0000 1694f700 -1 osd.4 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:51.765598+0000 front 2019-12-10T01:50:53.726284+0000 (oldest deadline 2019-12-10T01:51:33.673962+0000) 2019-12-10T01:52:08.125 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:52:08.114+0000 1694f700 -1 osd.1 191 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:50:53.626331+0000 front 2019-12-10T01:50:51.975197+0000 (oldest deadline 2019-12-10T01:51:33.547340+0000) 2019-12-10T01:52:08.181 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:52:08.174+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 45sec e191 v191) 2019-12-10T01:52:08.999 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:08.989+0000 1694f700 -1 osd.2 192 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:10.066 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:10.033+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:11.019 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:11.018+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:12.055 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:12.052+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:13.025 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:13.023+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:13.321 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:18:34.385 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:52:13.343 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:18:34.406 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:52:13.374 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:18:34.437 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:52:13.385 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:18:34.447 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:52:13.403 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:18:34.561 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:52:13.405 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:18:34.563 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:52:13.465 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:18:34.528 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:52:13.501 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:18:34.564 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:52:13.502 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:18:34.660 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:52:14.055 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:14.054+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:15.041 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:15.036+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:16.065 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:16.062+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:17.026 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:17.025+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:18.053 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:18.052+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:19.081 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:19.080+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:20.105 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:20.104+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:21.069 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:21.067+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:22.070 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:22.069+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:23.122 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:23.121+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:24.101 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:24.099+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:25.085 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:25.083+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:26.047 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:26.045+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:27.040 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:27.038+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:28.071 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:28.070+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:29.050 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:29.049+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:30.023 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:30.022+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:31.024 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:31.024+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:32.045 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:32.043+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:33.099 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:33.096+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:34.143 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:34.140+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:35.133 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:35.123+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:36.096 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:36.093+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:37.113 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:37.108+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:38.160 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:38.153+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:39.133 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:39.129+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:40.179 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:40.177+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:41.177 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:41.164+0000 1694f700 -1 osd.2 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:42.140 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:42.137+0000 1694f700 -1 osd.2 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:43.177 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:43.176+0000 1694f700 -1 osd.2 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:44.227 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:44.226+0000 1694f700 -1 osd.2 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:45.282 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:45.276+0000 1694f700 -1 osd.2 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:46.289 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:46.288+0000 1694f700 -1 osd.2 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:47.309 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:47.307+0000 1694f700 -1 osd.2 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:48.307 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:48.304+0000 1694f700 -1 osd.2 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:48.890 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:48.854+0000 32788700 -1 osd.0 195 build_incremental_map_msg missing incremental map 195 2019-12-10T01:52:48.958 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:48.914+0000 32f89700 -1 osd.0 195 build_incremental_map_msg missing incremental map 195 2019-12-10T01:52:49.289 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:49.123+0000 35f8f700 -1 osd.0 195 build_incremental_map_msg missing incremental map 195 2019-12-10T01:52:49.439 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:49.284+0000 3378a700 -1 osd.0 195 build_incremental_map_msg missing incremental map 195 2019-12-10T01:52:49.514 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:49.305+0000 32f89700 -1 osd.0 195 build_incremental_map_msg missing incremental map 195 2019-12-10T01:52:49.516 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:52:49.308+0000 32f89700 -1 osd.0 195 build_incremental_map_msg missing incremental map 195 2019-12-10T01:52:49.786 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:49.700+0000 1694f700 -1 osd.2 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:50.810 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:50.709+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:51.832 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:51.762+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:52.798 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:52.796+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:53.850 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:53.832+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:54.874 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:54.872+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:55.861 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:55.859+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:56.838 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:56.837+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:57.883 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:57.880+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:58.914 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:58.912+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:52:59.908 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:52:59.907+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:00.890 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:00.888+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:01.895 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:01.894+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:02.943 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:02.940+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:03.901 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:03.899+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:04.884 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:04.882+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:05.907 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:05.905+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:06.875 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:06.873+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:06.877 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:06.876+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:07.541 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:07.534+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:07.908 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:07.900+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:07.908 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:07.904+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:08.008 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:08.001+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:08.551 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:08.545+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:08.885 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:08.883+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:08.885 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:08.884+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:09.025 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:09.018+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:09.554 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:09.548+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:09.688 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:09.682+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:09.936 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:09.934+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:09.936 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:09.935+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:10.037 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:10.030+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:10.522 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:10.515+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:10.593 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:10.586+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:10.649 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:10.642+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:10.956 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:10.954+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:10.958 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:10.956+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:11.007 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:11.002+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:11.550 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:11.544+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:11.551 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:11.544+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:11.607 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:11.601+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:11.983 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:11.980+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:12.050 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:11.982+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:12.051 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:11.981+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:12.586 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:12.579+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:12.594 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:12.587+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:12.624 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:12.616+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:12.976 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:12.974+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:12.978 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:12.977+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:13.030 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:13.025+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:13.607 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:13.600+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:13.638 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:13.631+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:13.654 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:13.648+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:14.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:13.982+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:14.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:13.983+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:14.059 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:14.053+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:14.648 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:14.641+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:14.651 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:14.643+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:14.685 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:14.678+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:14.940 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:14.938+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:14.941 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:14.939+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:15.022 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:15.015+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:15.630 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:15.624+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:15.655 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:15.649+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:15.736 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:15.729+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:15.953 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:15.943+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:15.954 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:15.946+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:15.984 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:15.979+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:16.592 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:16.586+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:16.681 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:16.675+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:16.694 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:16.688+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:16.969 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:16.963+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:16.973 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:16.970+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:16.973 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:16.972+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:17.572 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:17.566+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:17.721 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:17.715+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:17.734 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:17.728+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:18.006 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:18.000+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:18.011 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:18.008+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:18.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:18.009+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:18.542 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:18.534+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:18.627 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:18.623+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:18.694 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:18.688+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:18.705 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:18.699+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:18.987 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:18.984+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:18.987 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:18.985+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:19.055 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:19.049+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:19.456 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:19.453+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:19.551 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:19.545+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:19.631 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:19.628+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:19.690 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:19.684+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:19.718 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:19.712+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:20.025 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:20.022+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:20.026 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:20.023+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:20.080 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:20.074+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:20.444 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:20.441+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:20.525 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:20.518+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:20.667 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:20.664+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:20.680 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:20.674+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:20.771 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:20.764+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:21.022 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:21.018+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:21.023 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:21.021+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:21.090 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:21.084+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:21.437 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:21.432+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:21.574 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:21.567+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:21.695 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:21.691+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:21.716 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:21.708+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:21.746 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:21.739+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:22.076 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:22.069+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:22.076 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:22.071+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:22.094 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:22.088+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:22.412 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:22.409+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:22.554 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:22.547+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:22.650 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:22.647+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:22.759 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:22.753+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:22.790 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:22.784+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:23.046 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:23.040+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:23.047 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:23.042+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:23.135 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:23.129+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:23.428 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:23.426+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:23.572 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:23.566+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:23.698 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:23.695+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:23.734 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:23.726+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:23.795 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:23.789+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:24.058 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:24.055+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:24.058 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:24.056+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:24.115 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:24.110+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:24.423 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:24.421+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:24.590 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:24.584+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:24.710 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:24.704+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:24.748 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:24.741+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:24.752 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:24.746+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:25.015 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:25.011+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:25.018 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:25.016+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:25.129 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:25.120+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:25.467 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:25.464+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:25.470 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:25.468+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:25.551 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:25.546+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:25.763 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:25.757+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:25.794 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:25.781+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:25.798 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:25.795+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:26.003 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:25.999+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:26.003 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:26.001+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:26.118 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:26.112+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:26.520 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:26.514+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:26.521 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:26.518+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:26.530 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:26.524+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:26.739 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:26.732+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:26.760 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:26.754+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:26.783 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:26.778+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:26.967 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:26.961+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:26.967 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:26.964+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:27.094 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:27.089+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:27.565 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:27.561+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:27.568 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:27.565+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:27.574 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:27.565+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:27.696 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:27.690+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:27.751 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:27.744+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:27.805 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:27.801+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:28.017 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:28.011+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:28.018 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:28.014+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:28.059 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:28.050+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:28.573 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:28.566+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:28.612 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:28.583+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:28.612 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:28.586+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:28.742 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:28.734+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:28.769 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:28.762+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:28.806 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:28.803+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:29.019 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:29.016+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:29.019 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:29.017+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:29.043 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:29.037+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:29.574 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:29.567+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:29.632 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:29.629+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:29.633 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:29.631+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:29.700 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:29.694+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:29.820 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:29.810+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:29.833 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:29.829+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:29.999 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:29.996+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:30.000 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:29.997+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:30.018 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:30.012+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:30.624 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:30.618+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:30.667 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:30.663+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:30.668 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:30.665+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:30.730 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:30.723+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:30.775 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:30.769+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:30.863 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:30.851+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:30.976 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:30.972+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:30.976 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:30.974+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:31.060 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:31.054+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:31.623 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:31.616+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:31.654 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:31.640+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:31.655 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:31.643+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:31.706 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:31.699+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:31.826 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:31.819+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:31.909 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:31.900+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:32.017 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:32.014+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:32.020 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:32.018+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:32.119 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:32.111+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:32.648 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:32.640+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:32.674 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:32.670+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:32.675 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:32.672+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:32.758 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:32.752+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:32.785 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:32.778+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:32.876 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:32.873+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:32.992 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:32.987+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:32.995 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:32.992+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:33.120 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:33.108+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:33.670 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:33.665+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:33.672 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:33.665+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:33.672 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:33.669+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:33.760 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:33.754+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:33.778 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:33.772+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:33.928 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:33.925+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:34.017 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:34.012+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:34.017 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:34.014+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:34.124 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:34.111+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:34.637 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:34.630+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:34.658 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:34.654+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:34.659 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:34.656+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:34.813 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:34.807+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:34.821 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:34.814+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:34.962 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:34.958+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:35.055 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:35.052+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:35.057 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:35.055+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:35.085 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:35.079+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:35.663 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:35.656+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:35.676 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:35.669+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:35.678 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:35.671+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:35.772 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:35.766+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:35.817 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:35.807+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:35.970 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:35.966+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:36.033 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:36.028+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:36.033 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:36.030+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:36.109 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:36.103+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:36.667 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:36.663+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:36.671 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:36.669+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:36.691 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:36.685+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:36.773 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:36.767+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:36.866 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:36.858+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:36.982 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:36.978+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:37.071 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:37.065+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:37.085 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:37.081+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:37.086 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:37.084+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:37.645 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:37.641+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:37.646 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:37.643+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:37.707 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:37.701+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:37.812 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:37.806+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:37.859 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:37.853+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:37.942 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:37.939+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:38.055 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:38.050+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:38.083 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:38.080+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:38.086 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:38.084+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:38.619 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:53:38.612+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e197 v197) 2019-12-10T01:53:38.685 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:38.678+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:38.687 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:38.682+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:38.688 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:38.685+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:38.866 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:38.861+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:38.867 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:38.860+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:38.904 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:38.900+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:39.064 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:39.060+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:39.064 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:39.062+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:39.110 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:39.105+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:39.670 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:39.658+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:39.701 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:39.694+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:39.701 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:39.695+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:39.859 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:39.854+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:39.917 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:39.911+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:39.952 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:39.948+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:40.045 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:40.040+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:40.046 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:40.043+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:40.162 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:40.156+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:40.707 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:40.701+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:40.746 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:40.742+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:40.746 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:40.744+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:40.883 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:40.877+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:40.913 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:40.906+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:40.940 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:40.935+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:41.091 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:41.086+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:41.092 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:41.089+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:41.207 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:41.200+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:41.670 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:41.663+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:41.742 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:41.736+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:41.743 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:41.737+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:41.875 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:41.868+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:41.885 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:41.877+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:41.937 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:41.935+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:42.046 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:42.043+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:42.053 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:42.045+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:42.206 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:42.200+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:42.641 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:42.632+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:42.772 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:42.768+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:42.775 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:42.773+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:42.833 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:42.826+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:42.891 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:42.885+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:42.979 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:42.976+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:43.053 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:43.047+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:43.053 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:43.050+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:43.188 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:43.181+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:43.608 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:43.602+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:43.636 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:53:43.629+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e197 v197) 2019-12-10T01:53:43.824 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:43.821+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:43.825 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:43.822+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:43.874 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:43.867+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:43.944 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:43.937+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:44.023 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:44.019+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:44.040 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:44.037+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:44.041 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:44.038+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:44.211 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:44.202+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:44.581 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:44.574+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:44.788 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:44.784+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:44.791 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:44.789+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:44.915 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:44.908+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:44.985 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:44.978+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:45.018 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:45.014+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:45.019 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:45.016+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:45.064 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:45.060+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:45.221 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:45.206+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:45.631 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:45.626+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:45.840 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:45.835+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:45.841 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:45.838+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:45.880 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:45.873+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:45.966 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:45.960+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:46.028 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:46.024+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:46.038 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:46.035+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:46.039 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:46.037+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:46.261 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:46.255+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:46.641 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:46.635+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:46.835 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:46.829+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:46.887 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:46.885+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:46.889 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:46.886+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:46.964 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:46.955+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:47.041 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:47.036+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:47.041 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:47.038+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:47.041 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:47.036+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:47.262 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:47.254+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:47.605 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:47.598+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:47.832 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:47.826+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:47.885 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:47.882+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:47.885 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:47.884+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:47.958 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:47.952+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:48.087 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:48.085+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:48.089 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:48.088+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:48.091 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:48.090+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:48.276 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:48.270+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:48.590 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:48.583+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:48.656 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:53:48.649+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e197 v197) 2019-12-10T01:53:48.825 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:48.819+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:48.901 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:48.900+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:48.903 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:48.902+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:48.979 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:48.970+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:49.109 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:49.105+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:49.114 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:49.112+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:49.117 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:49.113+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:49.273 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:49.267+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:49.620 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:49.614+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:49.803 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:49.797+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:49.904 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:49.901+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:49.905 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:49.904+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:49.979 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:49.973+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:50.073 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:50.068+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:50.111 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:50.108+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:50.112 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:50.110+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:50.240 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:50.235+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:50.668 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:50.662+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:50.769 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:50.763+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:50.946 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:50.941+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:50.947 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:50.944+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:50.991 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:50.985+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:51.048 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:51.045+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:51.122 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:51.119+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:51.124 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:51.123+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:51.288 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:51.282+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:51.685 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:51.678+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:51.744 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:51.738+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:51.971 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:51.964+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:51.975 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:51.971+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:51.975 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:51.973+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:52.058 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:52.055+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:52.103 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:52.099+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:52.104 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:52.101+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:52.258 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:52.252+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:52.663 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:52.657+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:52.745 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:52.738+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:52.932 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:52.930+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:52.935 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:52.934+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:53.018 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:53.011+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:53.049 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:53.044+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:53.100 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:53.098+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:53.105 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:53.101+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:53.226 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:53.219+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:53.681 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:53:53.673+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e197 v197) 2019-12-10T01:53:53.689 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:53.683+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:53.715 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:53.709+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:53.959 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:53.957+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:53.960 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:53.958+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:53.999 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:53.994+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:54.102 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:54.099+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:54.124 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:54.123+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:54.126 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:54.124+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:54.199 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:54.193+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:54.693 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:54.686+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:54.762 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:54.756+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:54.936 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:54.934+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:54.936 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:54.935+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:55.051 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:55.045+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:55.084 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:55.080+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:55.138 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:55.136+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:55.139 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:55.138+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:55.232 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:55.226+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:55.675 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:55.668+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:55.807 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:55.801+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:55.979 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:55.976+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:55.979 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:55.978+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:56.072 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:56.069+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:56.083 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:56.077+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:56.100 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:56.097+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:56.101 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:56.098+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:56.193 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:56.188+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:56.629 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:56.623+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:56.782 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:56.775+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:57.005 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:57.001+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:57.006 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:57.004+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:57.070 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:57.064+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:57.084 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:57.077+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:57.137 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:57.135+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:57.140 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:57.138+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:57.163 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:57.156+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:57.671 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:57.665+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:57.748 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:57.742+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:57.997 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:57.996+0000 1694f700 -1 osd.3 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:52:45.014869+0000 (oldest deadline 2019-12-10T01:53:25.014869+0000) 2019-12-10T01:53:57.998 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:57.997+0000 1694f700 -1 osd.3 197 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:53:58.053 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:53:58.050+0000 1694f700 -1 osd.0 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:24.319333+0000 front 2019-12-10T01:52:24.320581+0000 (oldest deadline 2019-12-10T01:53:18.478286+0000) 2019-12-10T01:53:58.053 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:53:58.042+0000 1694f700 -1 osd.7 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.314943+0000 front 2019-12-10T01:52:25.317804+0000 (oldest deadline 2019-12-10T01:53:09.789123+0000) 2019-12-10T01:53:58.171 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:53:58.165+0000 1694f700 -1 osd.5 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.844688+0000 front 2019-12-10T01:52:25.843869+0000 (oldest deadline 2019-12-10T01:53:07.871947+0000) 2019-12-10T01:53:58.174 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:58.171+0000 1694f700 -1 osd.2 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.309855+0000 front 2019-12-10T01:52:25.467949+0000 (oldest deadline 2019-12-10T01:53:06.255043+0000) 2019-12-10T01:53:58.175 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:53:58.174+0000 1694f700 -1 osd.2 197 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:104465 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [getxattrs out=916b,stat out=16b,read 0~168] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e189) 2019-12-10T01:53:58.648 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:53:58.641+0000 1694f700 -1 osd.6 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:25.890572+0000 front 2019-12-10T01:52:25.710580+0000 (oldest deadline 2019-12-10T01:53:09.183058+0000) 2019-12-10T01:53:58.720 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:53:58.716+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e197 v197) 2019-12-10T01:53:58.773 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:53:58.757+0000 1694f700 -1 osd.4 197 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:52:22.381990+0000 front 2019-12-10T01:52:21.883711+0000 (oldest deadline 2019-12-10T01:53:07.168317+0000) 2019-12-10T01:53:59.234 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:53:59.073+0000 1694f700 -1 osd.3 198 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e195) 2019-12-10T01:54:30.669 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:20:51.731 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:54:30.681 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:20:51.745 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:54:30.685 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:20:51.842 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:54:30.696 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:20:51.756 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:54:30.697 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:20:51.851 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:54:30.698 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:20:51.761 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:54:30.865 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:54:30.827+0000 124b0700 -1 osd.1 202 build_incremental_map_msg missing incremental map 202 2019-12-10T01:54:30.928 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:20:52.086 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:54:30.967 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:20:52.030 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:54:30.986 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:20:52.049 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:54:31.438 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:54:31.408+0000 3578e700 -1 osd.1 202 build_incremental_map_msg missing incremental map 202 2019-12-10T01:54:37.307 INFO:teuthology.orchestra.run.smithi067.stderr:testFileCreate (test.functional.tests.TestFile) ... ok 2019-12-10T01:54:51.753 INFO:teuthology.orchestra.run.smithi067.stderr:testGetContentType (test.functional.tests.TestFile) ... ok 2019-12-10T01:54:52.840 INFO:teuthology.orchestra.run.smithi067.stderr:testGetOnFileThatDoesNotExist (test.functional.tests.TestFile) ... ok 2019-12-10T01:54:56.728 INFO:teuthology.orchestra.run.smithi067.stderr:testHead (test.functional.tests.TestFile) ... ok 2019-12-10T01:54:57.586 INFO:teuthology.orchestra.run.smithi067.stderr:testHeadOnFileThatDoesNotExist (test.functional.tests.TestFile) ... ok 2019-12-10T01:55:26.841 INFO:teuthology.orchestra.run.smithi067.stderr:testMetadataOnPost (test.functional.tests.TestFile) ... ok 2019-12-10T01:55:29.913 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:29.905+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:30.873 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:30.867+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:30.973 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:30.970+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:31.907 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:31.900+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:32.006 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:32.001+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:32.448 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:32.442+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:32.800 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:32.793+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:32.886 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:32.875+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:33.060 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:33.056+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:33.195 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:33.189+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:33.266 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:33.257+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:33.445 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:33.434+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:33.809 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:33.797+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:33.899 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:33.894+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:34.110 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:34.108+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:34.246 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:34.241+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:34.261 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:34.257+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:34.484 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:34.478+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:34.578 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:34.575+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:34.766 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:34.761+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:34.951 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:34.945+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:35.113 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:35.109+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:35.238 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:35.229+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:35.265 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:35.261+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:35.524 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:35.518+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:35.589 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:35.579+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:35.804 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:35.799+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:35.961 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:35.954+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:36.150 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:36.146+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:36.194 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:36.186+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:36.291 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:36.286+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:36.484 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:36.478+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:36.627 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:36.621+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:36.795 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:36.788+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:36.974 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:36.968+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:37.113 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:37.109+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:37.211 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:37.205+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:37.287 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:37.284+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:37.521 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:37.515+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:37.638 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:37.635+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:37.807 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:37.797+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:37.969 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:37.963+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:38.136 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:38.133+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:38.227 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:38.221+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:38.286 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:38.283+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:38.478 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:38.472+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:38.694 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:38.690+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:38.800 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:38.794+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:39.018 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:39.012+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:39.142 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:39.139+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:39.207 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:39.201+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:39.293 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:39.290+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:39.445 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:39.438+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:39.734 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:39.718+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:39.785 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:39.779+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:39.978 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:39.973+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:40.170 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:40.166+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:40.262 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:40.253+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:40.282 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:40.267+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:40.422 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:40.415+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:40.728 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:40.724+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:40.747 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:40.741+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:41.005 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:40.999+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:41.169 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:41.166+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:41.256 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:41.252+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:41.259 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:41.254+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:41.424 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:41.418+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:41.753 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:41.748+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:41.781 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:41.777+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:41.964 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:41.959+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:42.200 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:42.197+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:42.257 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:42.253+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:42.293 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:42.287+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:42.381 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:42.375+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:42.727 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:42.721+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:42.802 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:42.797+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:43.001 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:42.995+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:43.242 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:43.238+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:43.308 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:43.305+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:43.331 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:43.325+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:43.427 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:43.422+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:43.750 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:43.741+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:43.766 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:43.763+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:43.964 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:43.960+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:44.274 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:44.270+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:44.278 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:44.274+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:44.322 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:44.317+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:44.389 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:44.383+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:44.745 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:44.736+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:44.797 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:44.793+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:44.997 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:44.991+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:45.257 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:45.253+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:45.311 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:45.306+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:45.345 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:45.339+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:45.378 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:45.370+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:45.764 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:45.755+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:45.768 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:45.765+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:46.026 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:46.020+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:46.294 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:46.290+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:46.333 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:46.330+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:46.344 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:46.338+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:46.406 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:46.398+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:46.740 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:46.736+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:46.774 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:46.766+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:46.983 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:46.977+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:47.295 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:47.291+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:47.365 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:47.362+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:47.377 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:47.371+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:47.400 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:47.395+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:47.714 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:47.710+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:47.745 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:47.739+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:47.948 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:47.942+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:48.333 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:48.310+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:48.357 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:48.352+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:48.400 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:48.397+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:48.417 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:48.412+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:48.703 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:48.698+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:48.737 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:48.731+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:48.948 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:48.943+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:49.318 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:49.315+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:49.365 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:49.359+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:49.380 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:49.377+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:49.458 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:49.449+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:49.699 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:49.694+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:49.744 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:49.738+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:49.949 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:49.944+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:50.329 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:50.326+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:50.352 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:50.346+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:50.354 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:50.351+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:50.439 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:50.431+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:50.654 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:50.652+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:50.794 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:50.786+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:50.993 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:55:50.987+0000 1694f700 -1 osd.5 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.104778+0000 front 2019-12-10T01:54:49.100863+0000 (oldest deadline 2019-12-10T01:55:29.768084+0000) 2019-12-10T01:55:51.348 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:51.341+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:51.362 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:51.359+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:51.403 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:51.387+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:51.475 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:51.470+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:51.656 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:51.653+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:51.789 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:51.782+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:52.329 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:52.323+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:52.383 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:52.380+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:52.384 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:52.381+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:52.461 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:52.452+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:52.653 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:52.649+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:52.804 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:52.798+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:53.326 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:53.319+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:53.339 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:53.336+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:53.403 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:53.399+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:53.504 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:53.498+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:53.653 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:53.649+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:53.762 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:53.756+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:54.281 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:54.274+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:54.375 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:54.371+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:54.414 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:54.410+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:54.552 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:54.539+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:54.628 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:54.624+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:54.803 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:54.798+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:55.262 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:55.256+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:55.383 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:55.379+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:55.396 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:55.392+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:55.600 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:55.595+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:55.639 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:55.635+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:55.821 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:55.800+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:56.247 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:56.242+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:56.361 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:56.357+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:56.422 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:56.419+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:56.583 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:56.577+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:56.669 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:56.663+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:56.772 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:56.767+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:57.275 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:57.268+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:57.381 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:57.375+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:57.414 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:57.410+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:57.553 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:57.547+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:57.631 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:57.624+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:57.783 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:57.777+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:58.305 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:58.299+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:58.361 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:58.358+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:58.409 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:58.404+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:58.530 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:58.524+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:58.604 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:58.600+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:58.743 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:58.737+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:55:59.273 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:55:59.267+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:55:59.400 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:55:59.396+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:55:59.421 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:55:59.417+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:55:59.556 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:55:59.549+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:55:59.613 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:55:59.608+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:55:59.740 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:55:59.734+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:00.275 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:00.270+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:00.412 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:56:00.409+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:56:00.463 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:56:00.459+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:56:00.535 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:56:00.529+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:56:00.595 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:00.586+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:56:00.742 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:56:00.735+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:01.280 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:01.273+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:01.385 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:56:01.378+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:56:01.461 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:56:01.458+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:56:01.574 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:56:01.567+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:56:01.592 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:01.588+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:56:01.791 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:56:01.780+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:02.266 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:02.261+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:02.342 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:56:02.339+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:56:02.474 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:56:02.471+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:56:02.589 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:56:02.582+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:56:02.616 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:02.612+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:56:02.832 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:56:02.825+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:03.301 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:03.294+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:03.385 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:56:03.380+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:56:03.490 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:56:03.488+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:56:03.597 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:56:03.591+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:56:03.637 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:03.625+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:56:03.840 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:56:03.834+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:04.073 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:56:04.072+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 40sec e202 v202) 2019-12-10T01:56:04.302 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:04.293+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:04.399 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:56:04.396+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:56:04.472 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:56:04.469+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:56:04.652 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:56:04.646+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:56:04.653 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:04.651+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:56:04.833 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:56:04.828+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:05.302 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:05.297+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:05.427 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:56:05.424+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:56:05.476 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:56:05.473+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:56:05.611 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:05.608+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:56:05.616 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:56:05.611+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:56:05.836 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:56:05.830+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:06.275 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:06.271+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:06.419 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:56:06.417+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:56:06.496 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:56:06.490+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:56:06.579 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:56:06.573+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:56:06.614 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:06.613+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:56:06.882 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:56:06.877+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:07.303 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:07.296+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:07.442 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:56:07.441+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:56:07.490 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:56:07.489+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:56:07.592 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:07.590+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:56:07.596 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:56:07.588+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:56:07.939 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:56:07.931+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:08.303 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:08.297+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:08.405 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:56:08.402+0000 1694f700 -1 osd.1 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.560143+0000 front 2019-12-10T01:54:48.839088+0000 (oldest deadline 2019-12-10T01:55:33.248088+0000) 2019-12-10T01:56:08.462 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:56:08.459+0000 1694f700 -1 osd.2 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:45.107783+0000 front 2019-12-10T01:54:45.108870+0000 (oldest deadline 2019-12-10T01:55:30.289368+0000) 2019-12-10T01:56:08.581 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:56:08.575+0000 1694f700 -1 osd.4 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:47.414881+0000 front 2019-12-10T01:54:47.426550+0000 (oldest deadline 2019-12-10T01:55:32.676323+0000) 2019-12-10T01:56:08.591 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:08.589+0000 1694f700 -1 osd.0 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.523270+0000 front 2019-12-10T01:54:48.521761+0000 (oldest deadline 2019-12-10T01:55:33.802005+0000) 2019-12-10T01:56:08.932 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:56:08.926+0000 1694f700 -1 osd.7 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:49.877560+0000 front 2019-12-10T01:54:49.854803+0000 (oldest deadline 2019-12-10T01:55:32.695902+0000) 2019-12-10T01:56:09.168 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:56:09.165+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 40sec e202 v202) 2019-12-10T01:56:09.353 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:56:09.348+0000 1694f700 -1 osd.6 202 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:54:48.059289+0000 front 2019-12-10T01:54:48.093826+0000 (oldest deadline 2019-12-10T01:55:32.102293+0000) 2019-12-10T01:56:40.298 INFO:teuthology.orchestra.run.smithi067.stderr:testMetadataOnPut (test.functional.tests.TestFile) ... ok 2019-12-10T01:56:49.843 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:49.726+0000 32788700 -1 osd.0 206 build_incremental_map_msg missing incremental map 206 2019-12-10T01:56:49.868 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:49.767+0000 32788700 -1 osd.0 206 build_incremental_map_msg missing incremental map 206 2019-12-10T01:56:49.869 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:49.774+0000 31f87700 -1 osd.0 206 build_incremental_map_msg missing incremental map 206 2019-12-10T01:56:49.957 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:49.825+0000 37792700 -1 osd.0 206 build_incremental_map_msg missing incremental map 206 2019-12-10T01:56:50.268 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:50.109+0000 32f89700 -1 osd.0 206 build_incremental_map_msg missing incremental map 206 2019-12-10T01:56:50.348 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:56:50.128+0000 32f89700 -1 osd.0 206 build_incremental_map_msg missing incremental map 206 2019-12-10T01:57:06.913 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:06.908+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:07.184 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:07.177+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:07.539 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:07.533+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:07.904 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:07.898+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:08.237 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:08.219+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:08.556 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:08.548+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:08.905 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:08.897+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:08.915 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:08.913+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:09.217 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:09.209+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:09.562 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:09.559+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:09.580 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:09.573+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:09.877 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:09.872+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:09.940 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:09.931+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:10.241 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:10.235+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:10.564 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:10.559+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:10.578 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:10.571+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:10.865 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:10.862+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:10.964 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:10.959+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:11.222 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:11.216+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:11.590 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:11.587+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:11.633 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:11.620+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:11.634 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:11.625+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:11.875 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:11.873+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:11.996 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:11.990+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:12.191 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:12.186+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:12.631 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:12.628+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:12.665 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:12.658+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:12.666 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:12.661+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:12.927 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:12.923+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:13.045 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:13.038+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:13.212 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:13.206+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:13.640 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:13.637+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:13.688 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:13.682+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:13.689 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:13.684+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:13.907 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:13.903+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:14.077 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:14.070+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:14.185 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:14.166+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:14.624 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:14.621+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:14.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:14.677+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:14.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:14.680+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:14.878 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:14.872+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:15.107 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:15.101+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:15.201 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:15.195+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:15.625 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:15.622+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:15.660 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:15.653+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:15.661 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:15.656+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:15.837 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:15.831+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:16.076 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:16.069+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:16.222 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:16.216+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:16.658 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:16.653+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:16.699 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:16.693+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:16.700 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:16.695+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:16.845 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:16.842+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:17.090 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:17.084+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:17.188 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:17.180+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:17.656 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:17.653+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:17.683 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:17.676+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:17.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:17.679+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:17.824 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:17.821+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:18.080 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:18.073+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:18.195 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:18.186+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:18.644 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:18.639+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:18.715 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:18.708+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:18.716 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:18.711+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:18.838 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:18.834+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:19.116 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:19.110+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:19.174 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:19.167+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:19.620 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:19.618+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:19.741 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:19.735+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:19.743 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:19.737+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:19.888 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:19.879+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:20.117 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:20.109+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:20.182 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:20.176+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:20.600 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:20.598+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:20.731 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:20.725+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:20.734 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:20.730+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:20.903 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:20.899+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:21.148 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:21.145+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:21.162 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:21.150+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:21.166 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:21.160+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:21.557 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:21.553+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:21.567 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:21.560+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:21.737 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:21.731+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:21.740 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:21.736+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:21.868 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:21.863+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:22.121 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:22.118+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:22.137 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:22.130+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:22.180 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:22.173+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:22.525 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:22.521+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:22.529 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:22.523+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:22.720 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:22.713+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:22.721 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:22.715+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:22.887 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:22.882+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:23.111 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:23.109+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:23.143 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:23.135+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:23.207 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:23.201+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:23.493 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:23.490+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:23.537 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:23.530+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:23.768 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:23.762+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:23.769 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:23.764+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:23.922 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:23.918+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:24.154 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:24.153+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:24.197 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:24.190+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:24.224 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:24.216+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:24.480 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:24.477+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:24.554 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:24.549+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:24.758 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:24.752+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:24.759 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:24.755+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:24.909 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:24.902+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:25.186 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:25.183+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:25.225 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:25.219+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:25.268 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:25.260+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:25.486 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:25.483+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:25.533 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:25.527+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:25.781 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:25.774+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:25.782 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:25.777+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:25.929 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:25.912+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:26.191 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:26.185+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:26.195 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:26.189+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:26.240 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:26.235+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:26.269 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:26.263+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:26.481 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:26.475+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:26.519 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:26.513+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:26.771 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:26.766+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:26.774 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:26.770+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:26.923 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:26.918+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:27.220 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:27.217+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:27.221 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:27.219+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:27.222 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:27.217+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:27.293 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:27.285+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:27.489 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:27.486+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:27.505 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:27.499+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:27.792 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:27.786+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:27.795 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:27.791+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:27.953 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:27.951+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:28.194 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:28.190+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:28.194 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:28.192+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:28.198 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:28.192+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:28.270 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:28.260+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:28.478 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:28.475+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:28.548 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:28.534+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:28.802 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:28.796+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:28.803 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:28.798+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:28.980 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:28.975+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:29.159 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:29.154+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:29.160 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:29.157+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:29.179 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:29.173+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:29.252 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:29.246+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:29.499 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:29.493+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:29.518 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:29.514+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:29.803 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:29.796+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:29.803 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:29.799+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:29.957 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:29.952+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:30.156 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:30.151+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:30.157 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:30.153+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:30.159 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:30.153+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:30.269 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:30.263+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:30.456 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:30.450+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:30.552 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:30.549+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:30.807 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:30.801+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:30.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:30.803+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:30.980 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:30.975+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:31.119 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:31.115+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:31.120 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:31.118+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:31.134 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:31.129+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:31.292 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:31.285+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:31.426 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:31.420+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:31.523 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:31.519+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:31.764 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:31.757+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:31.768 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:31.760+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:31.946 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:31.942+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:32.138 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:32.134+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:32.138 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:32.137+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:32.156 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:32.151+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:32.265 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:32.259+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:32.428 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:32.422+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:32.558 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:32.556+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:32.728 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:32.721+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:32.740 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:32.724+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:32.920 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:32.915+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:33.174 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:33.171+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:33.176 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:33.175+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:33.190 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:33.185+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:33.243 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:33.236+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:33.453 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:33.447+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:33.550 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:33.547+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:33.696 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:33.690+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:33.697 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:33.693+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:33.973 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:33.970+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:34.178 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:34.175+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:34.179 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:34.177+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:34.225 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:34.218+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:34.233 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:34.227+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:34.433 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:34.427+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:34.563 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:34.561+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:34.743 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:34.736+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:34.743 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:34.739+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:34.931 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:34.928+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:35.173 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:35.169+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:35.174 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:35.172+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:35.222 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:35.215+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:35.284 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:35.275+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:35.470 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:35.464+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:35.609 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:35.606+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:35.732 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:35.725+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:35.733 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:35.728+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:35.909 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:35.906+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:36.192 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:36.188+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:36.193 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:36.191+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:36.237 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:36.230+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:36.262 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:36.256+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:36.441 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:36.432+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:36.620 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:36.616+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:36.699 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:36.692+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:36.700 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:36.695+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:36.932 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:36.928+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:37.228 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:37.222+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:37.245 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:37.237+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:37.245 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:37.240+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:37.290 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:37.284+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:37.469 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:37.462+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:37.607 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:37.604+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:37.746 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:37.740+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:37.749 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:37.745+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:37.899 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:37.895+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:38.196 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:38.190+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:38.239 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:38.236+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:38.240 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:38.239+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:38.313 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:38.306+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:38.433 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:38.427+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:38.588 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:38.585+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:38.767 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:38.761+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:38.768 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:38.763+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:38.885 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:38.883+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:39.202 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:39.196+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:39.219 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:39.215+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:39.219 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:39.218+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:39.331 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:39.325+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:39.390 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:39.384+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:39.498 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:57:39.495+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e208 v208) 2019-12-10T01:57:39.589 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:39.587+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:39.725 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:39.718+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:39.725 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:39.721+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:39.919 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:39.917+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:40.176 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:40.170+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:40.194 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:40.190+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:40.194 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:40.193+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:40.352 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:40.346+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:40.440 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:40.429+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:40.632 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:40.630+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:40.756 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:40.750+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:40.759 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:40.755+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:40.898 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:40.894+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:41.170 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:41.163+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:41.212 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:41.208+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:41.213 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:41.211+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:41.399 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:41.393+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:41.474 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:41.468+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:41.626 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:41.624+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:41.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:41.801+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:41.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:41.804+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:41.862 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:41.859+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:42.180 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:42.175+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:42.242 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:42.239+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:42.242 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:42.241+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:42.364 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:42.352+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:42.448 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:42.442+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:42.611 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:42.608+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:42.819 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:42.813+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:42.838 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:42.827+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:42.838 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:42.831+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:43.148 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:43.141+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:43.239 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:43.235+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:43.240 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:43.238+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:43.347 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:43.341+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:43.471 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:43.464+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:43.588 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:43.586+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:43.796 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:43.787+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:43.796 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:43.790+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:43.847 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:43.843+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:44.161 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:44.154+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:44.267 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:44.265+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:44.271 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:44.269+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:44.373 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:44.366+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:44.486 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:44.479+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:44.516 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:57:44.516+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e208 v208) 2019-12-10T01:57:44.587 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:44.583+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:44.816 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:44.809+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:44.817 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:44.812+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:44.832 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:44.829+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:45.225 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:45.208+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:45.262 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:45.259+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:45.263 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:45.261+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:45.380 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:45.374+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:45.452 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:45.440+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:45.589 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:45.587+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:45.881 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:45.870+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:45.882 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:45.878+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:45.883 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:45.873+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:46.187 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:46.181+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:46.253 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:46.249+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:46.254 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:46.251+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:46.347 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:46.340+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:46.439 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:46.433+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:46.608 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:46.604+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:46.845 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:46.839+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:46.846 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:46.841+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:46.912 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:46.909+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:47.192 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:47.186+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:47.251 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:47.248+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:47.253 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:47.251+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:47.356 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:47.350+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:47.440 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:47.433+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:47.642 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:47.639+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:47.847 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:47.841+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:47.850 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:47.846+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:47.932 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:47.928+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:48.192 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:48.185+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:48.272 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:48.266+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:48.272 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:48.269+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:48.357 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:48.349+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:48.415 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:48.409+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:48.693 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:48.689+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:48.839 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:48.831+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:48.839 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:48.834+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:48.971 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:48.965+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:49.180 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:49.173+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:49.233 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:49.230+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:49.236 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:49.234+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:49.356 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:49.349+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:49.397 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:49.391+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:49.530 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:57:49.527+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e208 v208) 2019-12-10T01:57:49.651 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:49.649+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:49.864 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:49.858+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:49.867 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:49.863+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:49.970 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:49.966+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:50.220 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:50.213+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:50.247 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:50.243+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:50.249 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:50.247+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:50.315 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:50.308+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:50.441 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:50.435+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:50.627 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:50.625+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:50.899 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:50.892+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:50.900 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:50.895+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:50.937 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:50.934+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:51.202 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:51.195+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:51.231 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:51.226+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:51.232 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:51.229+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:51.305 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:51.299+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:51.422 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:51.416+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:51.620 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:51.617+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:51.904 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:51.897+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:51.907 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:51.903+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:51.935 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:51.931+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:52.181 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:52.175+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:52.237 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:52.234+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:52.238 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:52.235+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:52.289 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:52.280+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:52.422 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:52.416+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:52.660 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:52.656+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:52.903 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:52.896+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:52.904 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:52.899+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:52.963 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:52.957+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:53.219 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:53.212+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:53.254 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:53.250+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:53.263 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:53.255+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:53.322 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:53.316+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:53.433 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:53.427+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:53.617 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:53.612+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:53.941 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:53.930+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:53.941 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:53.932+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:53.990 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:53.986+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:54.252 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:54.246+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:54.310 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:54.305+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:54.311 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:54.308+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:54.349 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:54.343+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:54.474 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:54.468+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:54.554 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:57:54.543+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e208 v208) 2019-12-10T01:57:54.624 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:54.622+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:54.913 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:54.903+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:54.913 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:54.907+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:54.947 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:54.943+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:55.253 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:55.247+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:55.343 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:55.340+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:55.343 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:55.341+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:55.373 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:55.367+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:55.490 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:55.484+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:55.654 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:55.651+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:55.887 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:55.879+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:55.889 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:55.882+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:55.998 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:55.994+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:56.265 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:56.258+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:56.308 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:56.304+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:56.309 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:56.306+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:56.372 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:56.365+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:56.538 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:56.533+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:56.643 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:56.638+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:56.864 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:56.856+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:56.868 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:56.859+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:56.955 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:56.951+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:57.274 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:57.267+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:57.297 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:57.291+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:57.298 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:57.294+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:57.341 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:57.334+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:57.571 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:57.565+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:57.615 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:57.611+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:57.847 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:57.840+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:57.847 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:57.843+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:57.998 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:57.993+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:58.258 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:58.253+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:58.279 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:58.275+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:58.280 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:58.277+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:58.364 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:58.357+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:58.586 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:58.579+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:58.620 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:58.617+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:58.832 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:58.828+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:58.834 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:58.830+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:58.985 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:58.981+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:57:59.231 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:57:59.224+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:57:59.280 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:59.273+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:57:59.280 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:57:59.276+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:57:59.410 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:57:59.404+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:57:59.561 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:57:59.556+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e208 v208) 2019-12-10T01:57:59.607 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:57:59.604+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:57:59.625 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:57:59.620+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:57:59.837 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:59.828+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:57:59.840 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:57:59.834+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:57:59.961 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:57:59.957+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:58:00.280 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:00.276+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:58:00.280 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:00.278+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:58:00.292 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:58:00.285+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:58:00.395 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:58:00.389+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:58:00.629 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:58:00.625+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:58:00.660 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:58:00.653+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:58:00.833 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:58:00.825+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:58:00.837 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:58:00.832+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:58:00.957 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:58:00.955+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:58:01.237 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:01.232+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:58:01.237 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:01.235+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:58:01.335 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:58:01.330+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:58:01.408 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:58:01.402+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:58:01.616 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:58:01.610+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:58:01.672 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:58:01.668+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:58:01.872 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:58:01.866+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:58:01.875 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:58:01.871+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:58:02.004 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:58:01.998+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:58:02.243 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:02.240+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:58:02.244 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:02.242+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:58:02.304 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:58:02.297+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:58:02.427 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:58:02.420+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:58:02.606 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:58:02.600+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:58:02.689 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:58:02.685+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:58:02.900 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:58:02.893+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:58:02.901 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:58:02.895+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:58:03.062 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:58:03.058+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:58:03.216 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:03.211+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:58:03.217 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:03.214+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:58:03.303 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:58:03.296+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:58:03.381 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:58:03.374+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:58:03.570 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:58:03.564+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:58:03.705 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:58:03.701+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:58:03.945 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:58:03.937+0000 1694f700 -1 osd.5 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.829377+0000 front 2019-12-10T01:56:24.011781+0000 (oldest deadline 2019-12-10T01:57:06.692422+0000) 2019-12-10T01:58:03.948 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:58:03.944+0000 1694f700 -1 osd.5 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:58:04.107 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:58:04.102+0000 1694f700 -1 osd.0 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:24.713871+0000 front 2019-12-10T01:56:24.747257+0000 (oldest deadline 2019-12-10T01:57:08.794855+0000) 2019-12-10T01:58:04.217 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:04.214+0000 1694f700 -1 osd.3 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T01:56:45.207700+0000 (oldest deadline 2019-12-10T01:57:25.207700+0000) 2019-12-10T01:58:04.218 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T01:58:04.215+0000 1694f700 -1 osd.3 208 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e206) 2019-12-10T01:58:04.336 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:58:04.329+0000 1694f700 -1 osd.4 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:23.509523+0000 front 2019-12-10T01:56:23.521795+0000 (oldest deadline 2019-12-10T01:57:06.253225+0000) 2019-12-10T01:58:04.351 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:58:04.344+0000 1694f700 -1 osd.6 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:22.433113+0000 front 2019-12-10T01:56:22.623690+0000 (oldest deadline 2019-12-10T01:57:06.743633+0000) 2019-12-10T01:58:04.529 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:58:04.522+0000 1694f700 -1 osd.7 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:21.963526+0000 front 2019-12-10T01:56:22.063264+0000 (oldest deadline 2019-12-10T01:57:21.394282+0000) 2019-12-10T01:58:04.633 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T01:58:04.631+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e208 v208) 2019-12-10T01:58:04.752 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:58:04.749+0000 1694f700 -1 osd.2 208 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T01:56:25.383305+0000 front 2019-12-10T01:56:25.390098+0000 (oldest deadline 2019-12-10T01:57:09.466118+0000) 2019-12-10T01:58:05.001 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:58:04.992+0000 1694f700 -1 osd.5 209 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105623 2.2s0 2.3e1de382 (undecoded) ondisk+write+known_if_redirected e205) 2019-12-10T01:58:25.775 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:24:46.839 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:58:25.788 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:24:46.852 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:58:25.793 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:24:46.857 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:58:25.801 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:24:46.864 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:58:25.813 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:24:46.972 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:58:25.823 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:24:46.982 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:58:26.077 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:24:47.140 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T01:58:26.084 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:24:47.147 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T01:58:26.118 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:24:47.277 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T01:58:30.865 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:58:30.852+0000 3578e700 -1 osd.1 213 build_incremental_map_msg missing incremental map 213 2019-12-10T01:58:31.987 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:58:31.858+0000 31786700 -1 osd.1 214 build_incremental_map_msg missing incremental map 214 2019-12-10T01:58:42.707 INFO:teuthology.orchestra.run.smithi067.stderr:testNameLimit (test.functional.tests.TestFile) ... ok 2019-12-10T01:58:42.867 INFO:teuthology.orchestra.run.smithi067.stderr:testNoContentLengthForPut (test.functional.tests.TestFile) ... ok 2019-12-10T01:58:55.399 INFO:teuthology.orchestra.run.smithi067.stderr:testPostOnFileThatDoesNotExist (test.functional.tests.TestFile) ... ok 2019-12-10T01:59:26.803 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:26.799+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:27.777 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:27.766+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:28.731 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:28.726+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:29.705 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:29.700+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:30.683 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:30.678+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:31.132 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:31.129+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:31.161 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:31.157+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:31.696 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:31.692+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:32.099 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:32.095+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:32.211 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:32.206+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:32.726 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:32.722+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:32.978 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:32.972+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:33.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:33.050+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:33.199 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:33.195+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:33.743 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:33.736+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:33.744 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:33.738+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:33.981 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:33.976+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:34.095 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:34.093+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:34.193 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:34.187+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:34.251 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:34.248+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:34.430 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T01:59:34.427+0000 1694f700 -1 osd.0 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.889540+0000 front 2019-12-10T01:58:47.903296+0000 (oldest deadline 2019-12-10T01:59:33.725772+0000) 2019-12-10T01:59:34.735 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:34.727+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:34.742 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:34.738+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:34.944 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:34.940+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:34.949 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:34.943+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:35.057 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:35.054+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:35.192 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:35.184+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:35.249 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:35.247+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:35.699 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:35.694+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:35.729 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:35.697+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:35.967 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:35.963+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:35.976 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:35.971+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:36.081 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:36.073+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:36.187 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:36.171+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:36.244 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:36.241+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:36.714 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:36.707+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:36.716 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:36.710+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:36.948 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:36.936+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:36.973 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:36.966+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:37.061 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:37.059+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:37.210 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:37.203+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:37.231 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:37.224+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:37.718 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:37.712+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:37.726 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:37.717+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:37.946 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:37.941+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:37.959 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:37.953+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:38.016 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:38.013+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:38.189 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:38.186+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:38.207 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:38.197+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:38.699 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:38.692+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:38.699 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:38.694+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:38.935 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:38.929+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:38.975 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:38.969+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:39.043 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:39.039+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:39.150 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:39.146+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:39.248 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:39.243+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:39.678 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:39.671+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:39.679 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:39.674+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:39.941 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:39.935+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:40.021 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:40.014+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:40.079 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:40.077+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:40.123 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:40.120+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:40.266 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:40.260+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:40.716 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:40.710+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:40.717 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:40.712+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:40.935 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:40.929+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:41.013 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:41.008+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:41.037 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:41.033+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:41.143 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:41.139+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:41.280 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:41.273+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:41.679 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:41.667+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:41.680 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:41.670+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:41.920 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:41.914+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:42.004 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:41.997+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:42.091 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:42.087+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:42.102 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:42.097+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:42.323 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:42.312+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:42.646 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:42.638+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:42.646 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:42.640+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:42.917 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:42.910+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:42.971 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:42.965+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:43.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:43.049+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:43.137 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:43.133+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:43.362 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:43.354+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:43.698 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:43.692+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:43.699 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:43.694+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:43.937 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:43.931+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:43.954 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:43.948+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:44.036 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:44.033+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:44.176 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:44.172+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:44.399 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:44.393+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:44.709 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:44.703+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:44.709 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:44.705+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:44.917 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:44.911+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:44.931 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:44.924+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:45.010 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:45.006+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:45.215 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:45.212+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:45.384 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:45.378+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:45.678 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:45.672+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:45.681 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:45.677+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:45.918 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:45.912+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:45.956 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:45.949+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:46.002 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:45.999+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:46.198 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:46.194+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:46.364 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:46.359+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:46.711 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:46.703+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:46.713 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:46.706+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:46.930 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:46.924+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:47.002 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:46.996+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:47.013 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:47.011+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:47.191 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:47.189+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:47.353 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:47.347+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:47.741 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:47.734+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:47.741 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:47.736+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:47.911 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:47.906+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:47.965 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:47.958+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:48.066 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:48.064+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:48.156 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:48.153+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:48.383 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:48.378+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:48.759 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:48.752+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:48.760 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:48.754+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:48.901 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:48.894+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:48.967 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:48.962+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:49.109 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:49.102+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:49.178 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:49.119+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:49.412 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:49.406+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:49.735 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:49.727+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:49.736 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:49.729+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:49.864 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:49.857+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:49.955 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:49.950+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:50.094 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:50.092+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:50.148 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:50.146+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:50.377 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:50.370+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:50.770 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:50.764+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:50.773 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:50.769+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:50.837 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:50.827+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:50.960 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:50.954+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:51.068 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:51.064+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:51.117 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:51.115+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:51.420 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:51.414+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:51.761 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:51.755+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:51.761 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:51.757+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:51.812 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:51.805+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:51.972 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:51.966+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:52.022 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:52.019+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:52.168 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:52.165+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:52.423 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:52.416+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:52.779 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:52.773+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:52.780 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:52.775+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:52.783 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:52.777+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:52.946 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:52.939+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:53.024 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:53.022+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:53.182 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:53.181+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:53.430 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:53.424+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:53.770 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:53.764+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:53.797 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:53.791+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:53.798 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:53.793+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:53.928 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:53.922+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:53.993 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:53.985+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:54.206 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:54.203+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:54.413 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:54.407+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:54.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:54.797+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:54.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:54.801+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:54.812 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:54.806+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:54.956 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:54.950+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:55.019 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:55.016+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:55.238 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:55.235+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:55.373 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:55.367+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:55.793 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:55.787+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:55.838 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:55.832+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:55.839 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:55.834+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:55.936 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:55.930+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:56.019 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:56.017+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:56.203 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:56.199+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:56.356 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:56.350+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:56.833 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:56.827+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:56.876 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:56.870+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:56.877 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:56.872+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:56.978 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:56.972+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:57.066 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:57.063+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:57.187 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:57.184+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:57.407 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:57.401+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:57.818 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:57.812+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:57.857 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:57.851+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:57.858 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:57.853+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:58.001 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:57.994+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:58.049 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:58.047+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:58.238 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:58.236+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:58.380 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:58.374+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:58.822 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:58.817+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:58.887 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:58.882+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:58.891 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:58.886+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T01:59:59.053 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T01:59:59.047+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T01:59:59.061 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T01:59:59.058+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T01:59:59.203 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T01:59:59.200+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T01:59:59.359 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T01:59:59.353+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T01:59:59.797 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T01:59:59.791+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T01:59:59.883 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:59.877+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T01:59:59.884 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T01:59:59.879+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:00.074 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:00.068+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:00.096 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:00.094+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:00.249 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:00.246+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:00.374 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:00.369+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:00.824 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:00.816+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:00.900 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:00.894+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:00.903 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:00.898+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:01.055 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:01.051+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:01.086 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:01.079+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:01.225 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:01.221+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:01.412 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:01.406+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:01.830 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:01.824+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:01.927 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:01.920+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:01.927 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:01.922+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:02.083 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:02.079+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:02.097 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:02.082+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:02.205 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:02.198+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:02.459 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:02.453+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:02.847 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:02.841+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:02.896 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:02.884+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:02.896 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:02.885+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:03.068 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:03.058+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:03.125 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:03.122+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:03.239 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:03.236+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:03.449 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:03.439+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:03.890 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:03.883+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:03.890 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:03.885+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:03.895 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:03.889+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:04.089 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:04.085+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:04.096 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:04.091+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:04.242 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:04.239+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:04.424 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:04.418+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:04.890 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:04.883+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:04.932 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:00:04.929+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 45sec e214 v214) 2019-12-10T02:00:04.941 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:04.935+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:04.942 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:04.937+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:05.098 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:05.092+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:05.108 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:05.104+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:05.217 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:05.213+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:05.470 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:05.465+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:05.936 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:05.931+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:05.945 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:05.935+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:05.945 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:05.938+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:06.078 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:06.072+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:06.103 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:06.101+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:06.249 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:06.245+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:06.461 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:06.455+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:06.926 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:06.920+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:06.994 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:06.988+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:06.995 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:06.990+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:07.103 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:07.096+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:07.106 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:07.102+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:07.228 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:07.225+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:07.446 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:07.439+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:07.885 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:07.879+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:07.949 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:07.944+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:07.952 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:07.948+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:08.061 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:08.056+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:08.159 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:08.155+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:08.276 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:08.273+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:08.432 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:08.426+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:08.893 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:08.887+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:09.004 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:08.998+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:09.005 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:09.000+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:09.106 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:09.100+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:09.161 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:00:09.157+0000 1694f700 -1 osd.2 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.922984+0000 front 2019-12-10T01:58:46.933107+0000 (oldest deadline 2019-12-10T01:59:31.007445+0000) 2019-12-10T02:00:09.260 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:09.257+0000 1694f700 -1 osd.1 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:46.615551+0000 front 2019-12-10T01:58:46.665967+0000 (oldest deadline 2019-12-10T01:59:30.606079+0000) 2019-12-10T02:00:09.429 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:00:09.423+0000 1694f700 -1 osd.7 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:48.555153+0000 front 2019-12-10T01:58:48.553554+0000 (oldest deadline 2019-12-10T01:59:33.817478+0000) 2019-12-10T02:00:09.860 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:09.853+0000 1694f700 -1 osd.6 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:49.476293+0000 front 2019-12-10T01:58:49.411343+0000 (oldest deadline 2019-12-10T01:59:34.477929+0000) 2019-12-10T02:00:09.971 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:09.965+0000 1694f700 -1 osd.5 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:50.072802+0000 front 2019-12-10T01:58:50.047817+0000 (oldest deadline 2019-12-10T01:59:33.446999+0000) 2019-12-10T02:00:09.972 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:09.967+0000 1694f700 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:10.011 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:00:10.002+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 45sec e214 v214) 2019-12-10T02:00:10.060 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:00:10.054+0000 1694f700 -1 osd.4 214 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T01:58:47.010960+0000 front 2019-12-10T01:58:47.045461+0000 (oldest deadline 2019-12-10T01:59:32.295126+0000) 2019-12-10T02:00:11.280 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:11.261+0000 1694f700 -1 osd.5 215 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:12.306 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:00:12.301+0000 1694f700 -1 osd.5 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:105864 2.12s0 2:4aad4348:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_e1bb02f77d604e5a889fac4dc58ec66b:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~10053 in=10053b,setxattr user.rgw.manifest (361) in=378b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e214) 2019-12-10T02:00:26.608 INFO:teuthology.orchestra.run.smithi067.stderr:testRangedGets (test.functional.tests.TestFile) ... ok 2019-12-10T02:00:35.204 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:26:56.267 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:35.262 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:26:56.326 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:35.265 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:26:56.329 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:35.268 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:26:56.331 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:35.270 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:26:56.429 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:35.273 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:26:56.432 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:35.388 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:26:56.547 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:35.410 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:26:56.473 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:35.522 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:26:56.586 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:45.632 INFO:teuthology.orchestra.run.smithi067.stderr:testSerialization (test.functional.tests.TestFile) ... ok 2019-12-10T02:00:48.343 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:48.307+0000 114ae700 -1 osd.6 219 build_incremental_map_msg missing incremental map 219 2019-12-10T02:00:48.366 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:48.316+0000 36790700 -1 osd.6 219 build_incremental_map_msg missing incremental map 219 2019-12-10T02:00:48.394 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:48.319+0000 36790700 -1 osd.6 219 build_incremental_map_msg missing incremental map 219 2019-12-10T02:00:48.599 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:48.432+0000 33f8b700 -1 osd.6 219 build_incremental_map_msg missing incremental map 219 2019-12-10T02:00:48.612 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:48.436+0000 31f87700 -1 osd.6 219 build_incremental_map_msg missing incremental map 219 2019-12-10T02:00:48.660 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:48.489+0000 37792700 -1 osd.6 219 build_incremental_map_msg missing incremental map 219 2019-12-10T02:00:50.516 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:27:11.580 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:50.538 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:27:11.601 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:50.565 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:27:11.723 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:50.566 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:27:11.725 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:50.579 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:27:11.642 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:50.580 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:27:11.643 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:50.616 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:27:11.775 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:50.618 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:27:11.681 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:50.648 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:27:11.711 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:00:57.579 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:57.515+0000 31f87700 -1 osd.1 221 build_incremental_map_msg missing incremental map 202 2019-12-10T02:00:57.989 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:57.977+0000 31f87700 -1 osd.1 221 build_incremental_map_msg missing incremental map 213 2019-12-10T02:00:57.989 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:00:57.978+0000 31f87700 -1 osd.1 221 build_incremental_map_msg missing incremental map 214 2019-12-10T02:00:59.405 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:00:59.390+0000 34f8d700 -1 osd.6 221 build_incremental_map_msg missing incremental map 219 2019-12-10T02:01:00.788 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:00.779+0000 3478c700 -1 osd.0 221 build_incremental_map_msg missing incremental map 182 2019-12-10T02:01:00.789 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:00.780+0000 3478c700 -1 osd.0 221 build_incremental_map_msg missing incremental map 195 2019-12-10T02:01:00.790 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:00.782+0000 3478c700 -1 osd.0 221 build_incremental_map_msg missing incremental map 206 2019-12-10T02:01:07.840 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:07.836+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:08.019 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:08.013+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:08.147 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:08.141+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:08.836 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:08.831+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:08.991 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:08.984+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:08.999 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:08.995+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:09.116 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:09.110+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:09.159 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:09.154+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:09.821 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:09.818+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:09.976 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:09.969+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:10.030 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:10.027+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:10.132 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:10.127+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:10.198 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:10.191+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:10.864 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:10.862+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:11.010 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:11.003+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:11.035 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:11.031+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:11.163 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:11.158+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:11.166 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:11.160+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:11.837 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:11.833+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:12.019 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:12.012+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:12.023 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:12.019+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:12.146 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:12.141+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:12.205 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:12.199+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:12.818 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:12.812+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:12.984 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:12.978+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:13.008 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:13.001+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:13.113 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:13.107+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:13.253 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:13.246+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:13.529 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:13.523+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:13.851 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:13.849+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:13.994 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:13.990+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:14.059 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:14.053+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:14.137 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:14.131+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:14.295 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:14.289+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:14.498 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:14.491+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:14.856 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:14.853+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:15.010 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:15.007+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:15.039 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:15.032+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:15.165 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:15.158+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:15.313 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:15.305+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:15.552 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:15.545+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:15.835 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:15.832+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:16.023 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:16.013+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:16.040 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:16.035+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:16.133 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:16.126+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:16.292 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:16.286+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:16.529 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:16.523+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:16.806 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:16.802+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:16.999 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:16.991+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:17.034 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:17.029+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:17.162 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:17.155+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:17.345 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:17.335+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:17.487 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:17.482+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:17.822 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:17.819+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:17.976 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:17.970+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:18.076 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:18.069+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:18.143 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:18.137+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:18.271 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:18.269+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:18.329 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:18.322+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:18.492 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:18.478+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:18.788 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:18.784+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:18.984 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:18.977+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:19.083 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:19.081+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:19.194 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:19.188+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:19.267 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:19.266+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:19.313 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:19.307+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:19.488 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:19.482+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:19.785 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:19.783+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:19.968 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:19.962+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:20.058 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:20.054+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:20.216 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:20.210+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:20.269 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:20.263+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:20.284 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:20.283+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:20.530 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:20.524+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:20.765 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:20.763+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:21.001 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:20.995+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:21.069 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:21.064+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:21.206 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:21.200+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:21.263 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:21.257+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:21.336 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:21.332+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:21.542 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:21.536+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:21.801 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:21.798+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:22.045 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:22.039+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:22.071 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:22.069+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:22.169 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:22.162+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:22.262 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:22.257+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:22.347 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:22.345+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:22.500 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:22.494+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:22.785 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:22.783+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:23.098 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:23.091+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:23.109 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:23.102+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:23.188 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:23.183+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:23.219 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:23.214+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:23.391 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:23.389+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:23.481 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:23.473+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:23.806 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:23.803+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:24.067 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:24.063+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:24.151 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:24.145+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:24.171 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:24.165+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:24.258 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:24.251+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:24.414 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:24.410+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:24.454 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:24.447+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:24.818 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:24.816+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:25.111 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:25.107+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:25.194 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:25.187+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:25.197 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:25.190+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:25.219 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:25.211+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:25.429 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:25.427+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:25.504 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:25.496+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:25.816 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:25.814+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:26.147 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:26.143+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:26.155 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:26.149+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:26.176 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:26.170+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:26.238 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:26.232+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:26.473 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:26.467+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:26.474 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:26.471+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:26.550 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:26.545+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:26.777 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:26.774+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:27.105 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:27.103+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:27.198 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:27.191+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:27.210 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:27.203+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:27.283 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:27.276+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:27.486 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:27.482+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:27.486 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:27.483+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:27.590 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:27.584+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:27.749 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:27.747+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:28.058 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:28.056+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:28.224 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:28.217+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:28.239 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:28.233+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:28.282 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:28.275+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:28.503 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:28.494+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:28.504 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:28.499+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:28.612 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:28.601+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:28.710 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:28.707+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:29.125 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:29.106+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:29.220 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:29.213+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:29.254 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:29.248+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:29.257 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:29.250+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:29.531 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:29.528+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:29.532 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:29.530+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:29.652 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:29.646+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:29.751 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:29.748+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:30.090 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:30.088+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:30.247 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:30.241+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:30.268 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:30.261+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:30.285 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:30.280+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:30.495 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:30.492+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:30.497 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:30.494+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:30.645 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:30.638+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:30.797 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:30.790+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:31.059 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:31.056+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:31.209 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:31.203+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:31.220 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:31.215+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:31.259 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:31.249+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:31.535 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:31.531+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:31.536 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:31.533+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:31.690 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:31.679+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:31.748 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:31.744+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:32.048 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:32.045+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:32.189 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:32.171+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:32.239 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:32.232+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:32.246 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:32.240+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:32.570 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:32.565+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:32.571 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:32.568+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:32.671 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:32.664+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:32.724 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:32.721+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:33.034 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:33.026+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:33.218 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:33.212+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:33.227 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:33.220+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:33.301 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:33.293+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:33.617 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:33.614+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:33.620 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:33.617+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:33.633 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:33.627+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:33.706 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:33.702+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:34.047 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:34.043+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:34.206 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:34.199+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:34.276 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:34.271+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:34.341 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:34.335+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:34.587 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:34.584+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:34.588 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:34.586+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:34.612 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:34.603+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:34.744 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:34.741+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:35.096 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:35.094+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:35.256 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:35.250+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:35.306 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:35.300+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:35.309 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:35.303+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:35.580 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:35.577+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:35.584 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:35.581+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:35.665 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:35.652+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:35.718 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:35.706+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:36.064 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:36.059+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:36.223 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:36.217+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:36.296 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:36.291+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:36.348 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:36.341+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:36.585 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:36.579+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:36.586 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:36.583+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:36.654 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:36.644+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:36.699 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:36.696+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:37.077 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:37.072+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:37.246 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:37.240+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:37.297 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:37.290+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:37.369 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:37.360+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:37.607 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:37.603+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:37.612 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:37.607+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:37.627 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:37.622+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:37.749 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:37.746+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:38.105 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:38.102+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:38.225 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:38.217+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:38.276 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:38.267+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:38.395 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:38.388+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:38.581 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:38.576+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:38.582 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:38.579+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:38.679 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:38.673+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:38.740 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:38.736+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:39.178 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:39.167+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:39.186 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:39.180+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:39.310 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:39.305+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:39.413 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:39.406+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:39.537 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:39.532+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:39.538 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:39.535+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:39.700 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:39.693+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:39.724 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:39.720+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:40.155 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:40.151+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:40.214 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:40.207+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:40.355 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:40.345+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:40.435 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:40.428+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:40.573 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:40.565+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:40.574 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:40.568+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:40.616 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:01:40.611+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e221 v221) 2019-12-10T02:01:40.715 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:40.709+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:40.773 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:40.770+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:41.148 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:41.145+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:41.200 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:41.192+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:41.359 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:41.354+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:41.448 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:41.441+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:41.549 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:41.546+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:41.551 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:41.549+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:41.741 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:41.738+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:41.744 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:41.738+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:42.103 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:42.100+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:42.163 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:42.156+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:42.364 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:42.358+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:42.451 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:42.444+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:42.573 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:42.567+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:42.574 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:42.569+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:42.710 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:42.705+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:42.756 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:42.752+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:43.128 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:43.125+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:43.164 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:43.149+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:43.326 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:43.320+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:43.462 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:43.456+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:43.565 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:43.557+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:43.565 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:43.560+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:43.736 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:43.730+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:43.748 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:43.744+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:44.118 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:44.115+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:44.211 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:44.204+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:44.308 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:44.302+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:44.497 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:44.484+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:44.589 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:44.585+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:44.593 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:44.590+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:44.722 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:44.716+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:44.786 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:44.782+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:45.097 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:45.093+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:45.181 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:45.174+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:45.314 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:45.308+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:45.487 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:45.480+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:45.609 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:45.606+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:45.613 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:45.610+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:45.622 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:01:45.620+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e221 v221) 2019-12-10T02:01:45.697 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:45.688+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:45.773 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:45.769+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:46.092 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:46.087+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:46.229 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:46.223+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:46.332 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:46.326+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:46.509 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:46.503+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:46.605 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:46.600+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:46.605 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:46.603+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:46.705 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:46.696+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:46.775 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:46.760+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:47.117 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:47.109+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:47.258 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:47.251+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:47.353 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:47.345+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:47.506 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:47.493+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:47.528 INFO:teuthology.orchestra.run.smithi067.stderr:testStackedOverwrite (test.functional.tests.TestFile) ... ok 2019-12-10T02:01:47.592 INFO:teuthology.orchestra.run.smithi067.stderr:testTooLongName (test.functional.tests.TestFile) ... ok 2019-12-10T02:01:47.616 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:47.612+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:47.617 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:47.614+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:47.713 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:47.708+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:47.729 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:47.726+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:48.092 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:48.088+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:48.294 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:48.287+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:48.355 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:48.349+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:48.489 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:48.478+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:48.590 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:48.585+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:48.695 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:48.591+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:48.716 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:48.711+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:48.727 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:48.721+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:49.113 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:49.110+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:49.320 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:49.302+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:49.320 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:49.308+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:49.497 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:49.489+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:49.607 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:49.604+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:49.610 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:49.608+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:49.635 INFO:teuthology.orchestra.run.smithi067.stderr:testZeroByteFile (test.functional.tests.TestFile) ... ok 2019-12-10T02:01:49.699 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:49.696+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:49.723 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:49.717+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:50.099 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:50.095+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:50.282 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:50.276+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:50.310 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:50.304+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:50.502 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:50.492+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:50.636 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:01:50.631+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e221 v221) 2019-12-10T02:01:50.651 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:50.647+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:50.656 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:50.653+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:50.663 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:50.658+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:50.777 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:50.771+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:51.144 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:51.140+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:51.302 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:51.296+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:51.314 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:51.307+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:51.503 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:51.496+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:51.662 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:51.659+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:51.664 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:51.662+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:51.684 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:51.681+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:51.777 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:51.771+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:52.109 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:52.106+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:52.331 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:52.324+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:52.351 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:52.345+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:52.554 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:52.548+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:52.649 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:52.644+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:52.826 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:52.794+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:52.827 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:52.674+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:52.828 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:52.682+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:53.111 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:53.107+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:53.331 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:53.325+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:53.385 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:53.379+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:53.531 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:53.524+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:53.693 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:53.688+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:53.694 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:53.690+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:53.697 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:53.693+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:53.787 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:53.782+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:54.078 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:54.074+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:54.301 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:54.293+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:54.400 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:54.393+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:54.509 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:54.492+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:54.670 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:54.667+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:54.673 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:54.670+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:54.695 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:54.692+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:54.823 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:54.816+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:55.131 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:55.127+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:55.351 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:55.344+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:55.391 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:55.384+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:55.473 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:55.466+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:55.650 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:01:55.646+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e221 v221) 2019-12-10T02:01:55.705 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:55.697+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:55.706 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:55.700+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:55.714 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:55.705+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:55.820 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:55.814+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:56.088 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:56.084+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:56.328 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:56.321+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:56.404 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:56.398+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:56.478 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:56.472+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:56.711 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:56.674+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:56.716 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:56.678+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:56.728 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:56.724+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:56.855 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:56.848+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:57.135 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:57.132+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:57.369 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:57.365+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:57.392 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:57.387+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:57.467 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:57.462+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:57.657 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:57.655+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:57.659 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:57.657+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:57.683 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:57.679+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:57.907 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:57.900+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:58.136 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:58.132+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:58.351 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:58.340+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:58.405 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:58.400+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:58.442 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:58.436+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:58.631 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:58.628+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:58.635 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:58.633+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:58.667 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:58.663+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:58.911 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:58.905+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:01:59.111 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:01:59.108+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:01:59.369 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:01:59.362+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:01:59.419 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:01:59.413+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:01:59.491 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:01:59.483+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:01:59.616 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:59.610+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:01:59.618 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:01:59.614+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:01:59.712 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:01:59.708+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:01:59.967 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:01:59.958+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:02:00.121 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:02:00.111+0000 1694f700 -1 osd.0 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.420830+0000 front 2019-12-10T02:00:22.676756+0000 (oldest deadline 2019-12-10T02:01:08.287643+0000) 2019-12-10T02:02:00.402 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:02:00.395+0000 1694f700 -1 osd.7 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.114751+0000 front 2019-12-10T02:00:22.157098+0000 (oldest deadline 2019-12-10T02:01:07.900752+0000) 2019-12-10T02:02:00.439 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:02:00.433+0000 1694f700 -1 osd.5 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:25.365452+0000 front 2019-12-10T02:00:25.367919+0000 (oldest deadline 2019-12-10T02:01:08.181983+0000) 2019-12-10T02:02:00.510 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:02:00.500+0000 1694f700 -1 osd.6 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.680907+0000 front 2019-12-10T02:00:22.682206+0000 (oldest deadline 2019-12-10T02:01:07.118589+0000) 2019-12-10T02:02:00.659 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:02:00.653+0000 1694f700 -1 osd.3 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:00:46.050398+0000 (oldest deadline 2019-12-10T02:01:26.050398+0000) 2019-12-10T02:02:00.661 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:02:00.657+0000 1694f700 -1 osd.3 221 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e219) 2019-12-10T02:02:00.732 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:02:00.726+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e221 v221) 2019-12-10T02:02:00.755 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:02:00.752+0000 1694f700 -1 osd.2 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:22.682503+0000 front 2019-12-10T02:00:22.472183+0000 (oldest deadline 2019-12-10T02:01:07.654011+0000) 2019-12-10T02:02:00.934 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:02:00.925+0000 1694f700 -1 osd.4 221 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:00:26.936915+0000 front 2019-12-10T02:00:26.996331+0000 (oldest deadline 2019-12-10T02:01:12.549862+0000) 2019-12-10T02:02:07.708 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:28:28.771 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:02:07.712 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:28:28.775 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:02:07.719 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:28:28.782 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:02:07.722 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:28:28.785 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:02:07.740 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:28:28.899 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:02:07.750 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:28:28.909 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:02:07.840 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:28:28.903 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:02:07.857 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:28:28.920 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:02:07.960 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:28:29.119 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:02:29.412 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:02:29.350+0000 31f87700 -1 osd.6 225 build_incremental_map_msg missing incremental map 225 2019-12-10T02:02:29.453 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:02:29.387+0000 31f87700 -1 osd.6 225 build_incremental_map_msg missing incremental map 225 2019-12-10T02:02:29.504 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:02:29.443+0000 35f8f700 -1 osd.6 225 build_incremental_map_msg missing incremental map 225 2019-12-10T02:02:29.532 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:02:29.452+0000 3578e700 -1 osd.6 225 build_incremental_map_msg missing incremental map 225 2019-12-10T02:02:29.570 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:02:29.516+0000 3578e700 -1 osd.6 225 build_incremental_map_msg missing incremental map 225 2019-12-10T02:02:31.501 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:02:31.490+0000 32788700 -1 osd.1 226 build_incremental_map_msg missing incremental map 226 2019-12-10T02:02:31.885 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:02:31.785+0000 31f87700 -1 osd.1 226 build_incremental_map_msg missing incremental map 226 2019-12-10T02:02:31.956 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:02:31.786+0000 31f87700 -1 osd.1 226 build_incremental_map_msg missing incremental map 226 2019-12-10T02:02:31.997 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:02:31.801+0000 3578e700 -1 osd.1 226 build_incremental_map_msg missing incremental map 226 2019-12-10T02:02:31.997 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:02:31.817+0000 3578e700 -1 osd.1 226 build_incremental_map_msg missing incremental map 226 2019-12-10T02:03:30.115 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:30.107+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:31.109 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:31.105+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:32.157 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:32.153+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:32.863 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:32.860+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:33.147 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:33.142+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:33.501 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:33.492+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:33.855 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:33.853+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:34.001 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:03:33.993+0000 1694f700 -1 osd.1 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:49.330941+0000 front 2019-12-10T02:02:49.242572+0000 (oldest deadline 2019-12-10T02:03:33.750347+0000) 2019-12-10T02:03:34.133 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:34.121+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:34.223 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:34.216+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:34.527 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:34.521+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:34.571 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:34.564+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:34.847 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:34.844+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:35.047 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:03:35.040+0000 1694f700 -1 osd.1 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:49.330941+0000 front 2019-12-10T02:02:49.242572+0000 (oldest deadline 2019-12-10T02:03:33.750347+0000) 2019-12-10T02:03:35.103 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:35.098+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:35.221 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:35.212+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:35.563 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:35.556+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:35.563 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:35.556+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:35.844 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:35.843+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:36.006 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:03:36.004+0000 1694f700 -1 osd.1 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:49.330941+0000 front 2019-12-10T02:02:49.242572+0000 (oldest deadline 2019-12-10T02:03:33.750347+0000) 2019-12-10T02:03:36.112 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:36.106+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:36.114 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:36.108+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:36.212 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:36.206+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:36.554 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:36.546+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:36.601 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:36.594+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:36.847 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:36.843+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:37.024 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:03:37.020+0000 1694f700 -1 osd.1 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:49.330941+0000 front 2019-12-10T02:02:49.242572+0000 (oldest deadline 2019-12-10T02:03:33.750347+0000) 2019-12-10T02:03:37.164 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:37.152+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:37.167 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:37.163+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:37.204 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:37.194+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:37.507 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:37.505+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:37.568 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:37.547+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:37.639 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:37.632+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:37.832 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:37.830+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:37.987 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:03:37.985+0000 1694f700 -1 osd.1 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:49.330941+0000 front 2019-12-10T02:02:49.242572+0000 (oldest deadline 2019-12-10T02:03:33.750347+0000) 2019-12-10T02:03:38.164 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:38.156+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:38.166 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:38.160+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:38.233 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:38.227+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:38.469 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:38.465+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:38.530 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:38.524+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:38.666 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:38.661+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:38.808 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:38.803+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:39.038 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:03:39.034+0000 1694f700 -1 osd.1 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:49.330941+0000 front 2019-12-10T02:02:49.242572+0000 (oldest deadline 2019-12-10T02:03:33.750347+0000) 2019-12-10T02:03:39.213 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:39.207+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:39.214 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:39.209+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:39.245 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:39.239+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:39.488 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:39.486+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:39.491 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:39.486+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:39.640 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:39.634+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:39.789 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:39.785+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:40.057 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:03:40.055+0000 1694f700 -1 osd.1 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:49.330941+0000 front 2019-12-10T02:02:49.242572+0000 (oldest deadline 2019-12-10T02:03:33.750347+0000) 2019-12-10T02:03:40.233 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:40.227+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:40.238 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:40.229+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:40.294 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:40.286+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:40.523 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:40.517+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:40.537 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:40.531+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:40.608 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:40.602+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:40.836 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:40.833+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:41.286 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:41.278+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:41.287 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:41.280+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:41.305 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:41.299+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:41.543 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:41.537+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:41.553 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:41.549+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:41.610 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:41.603+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:41.858 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:41.855+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:42.269 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:42.259+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:42.269 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:42.261+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:42.284 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:42.279+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:42.547 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:42.541+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:42.595 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:42.591+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:42.651 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:42.642+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:42.849 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:42.842+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:43.268 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:43.262+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:43.271 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:43.267+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:43.315 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:43.309+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:43.517 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:43.511+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:43.627 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:43.616+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:43.657 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:43.652+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:43.882 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:43.879+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:44.286 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:44.280+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:44.327 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:44.320+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:44.328 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:44.323+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:44.509 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:44.504+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:44.600 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:44.597+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:44.654 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:44.645+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:44.886 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:44.882+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:45.284 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:45.279+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:45.325 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:45.319+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:45.326 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:45.321+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:45.483 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:45.474+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:45.589 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:45.587+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:45.620 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:45.614+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:45.916 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:45.912+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:46.292 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:46.286+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:46.362 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:46.356+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:46.363 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:46.358+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:46.509 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:46.504+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:46.561 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:46.558+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:46.635 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:46.629+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:46.906 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:46.901+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:47.336 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:47.331+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:47.367 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:47.361+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:47.367 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:47.363+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:47.501 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:47.495+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:47.544 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:47.541+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:47.636 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:47.629+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:47.922 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:47.919+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:48.344 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:48.327+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:48.406 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:48.397+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:48.406 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:48.400+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:48.457 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:48.451+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:48.532 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:48.530+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:48.614 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:48.605+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:48.952 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:48.949+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:49.347 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:49.342+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:49.423 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:49.417+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:49.435 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:49.429+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:49.436 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:49.431+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:49.518 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:49.514+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:49.605 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:49.599+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:49.981 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:49.977+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:50.388 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:50.383+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:50.433 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:50.427+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:50.434 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:50.429+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:50.462 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:50.457+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:50.515 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:50.511+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:50.643 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:50.636+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:51.014 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:51.010+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:51.381 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:51.375+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:51.449 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:51.443+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:51.452 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:51.448+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:51.493 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:51.482+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:51.509 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:51.503+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:51.642 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:51.638+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:52.061 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:52.057+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:52.428 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:52.421+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:52.446 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:52.441+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:52.449 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:52.445+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:52.457 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:52.454+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:52.495 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:52.489+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:52.603 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:52.598+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:53.077 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:53.073+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:53.420 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:53.415+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:53.437 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:53.432+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:53.443 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:53.435+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:53.469 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:53.466+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:53.475 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:53.469+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:53.598 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:53.593+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:54.055 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:54.051+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:54.411 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:54.405+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:54.439 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:54.433+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:54.440 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:54.435+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:54.478 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:54.472+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:54.478 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:54.474+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:54.566 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:54.560+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:55.066 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:55.062+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:55.378 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:55.372+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:55.444 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:55.439+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:55.445 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:55.439+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:55.447 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:55.443+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:55.528 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:55.522+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:55.615 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:55.610+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:56.062 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:56.059+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:56.401 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:56.395+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:56.440 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:56.434+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:56.441 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:56.436+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:56.446 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:56.442+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:56.573 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:56.566+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:56.600 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:56.594+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:57.051 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:57.046+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:57.397 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:57.390+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:57.400 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:57.393+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:57.402 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:57.398+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:57.429 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:57.423+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:57.599 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:57.594+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:57.604 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:57.598+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:58.077 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:58.073+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:58.397 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:58.390+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:58.400 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:58.392+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:58.400 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:58.395+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:58.442 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:58.439+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:58.595 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:58.589+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:03:58.649 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:58.639+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:59.041 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:03:59.033+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:03:59.362 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:03:59.356+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:03:59.383 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:59.378+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:03:59.386 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:03:59.382+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:03:59.458 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:03:59.454+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:03:59.632 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:03:59.626+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:03:59.634 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:03:59.628+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:00.084 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:00.080+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:00.384 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:00.378+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:00.420 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:00.415+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:00.421 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:00.415+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:00.422 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:00.417+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:00.635 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:00.628+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:00.639 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:00.632+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:01.096 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:01.092+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:01.345 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:01.333+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:01.422 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:01.420+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:01.440 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:01.432+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:01.440 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:01.434+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:01.605 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:01.599+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:01.693 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:01.687+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:02.062 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:02.057+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:02.344 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:02.339+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:02.428 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:02.425+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:02.473 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:02.464+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:02.473 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:02.465+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:02.611 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:02.606+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:02.709 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:02.703+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:03.037 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:03.034+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:03.326 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:03.320+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:03.456 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:03.453+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:03.509 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:03.503+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:03.509 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:03.505+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:03.582 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:03.575+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:03.673 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:03.667+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:04.088 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:04.085+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:04.366 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:04.361+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:04.434 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:04.431+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:04.495 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:04.486+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:04.501 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:04.497+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:04.634 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:04.625+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:04.668 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:04.662+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:05.094 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:05.090+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:05.338 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:05.332+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:05.415 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:05.410+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:05.532 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:05.527+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:05.533 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:05.528+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:05.589 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:05.583+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:05.630 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:05.624+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:06.080 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:06.076+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:06.140 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:04:06.138+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 2 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 47sec e226 v226) 2019-12-10T02:04:06.343 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:06.337+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:06.417 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:06.413+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:06.493 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:06.487+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:06.494 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:06.489+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:06.634 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:06.629+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:06.641 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:06.636+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:07.090 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:07.086+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:07.301 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:07.296+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:07.434 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:07.431+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:07.472 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:07.465+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:07.472 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:07.467+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:07.622 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:07.617+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:07.685 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:07.680+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:08.098 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:08.095+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:08.273 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:08.267+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:08.435 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:08.430+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:08.438 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:08.434+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:08.439 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:08.435+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:08.609 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:08.603+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:08.733 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:08.724+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:09.125 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:09.122+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:09.265 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:09.259+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:09.435 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:09.429+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:09.436 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:09.431+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:09.443 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:09.441+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:09.656 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:09.650+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:09.700 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:09.695+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:10.123 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:10.120+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:10.241 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:10.236+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:10.414 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:10.408+0000 1694f700 -1 osd.6 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:50.968276+0000 front 2019-12-10T02:02:50.559233+0000 (oldest deadline 2019-12-10T02:03:35.749860+0000) 2019-12-10T02:04:10.414 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:10.410+0000 1694f700 -1 osd.6 226 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:10.446 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:04:10.443+0000 1694f700 -1 osd.2 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.394623+0000 front 2019-12-10T02:02:51.400385+0000 (oldest deadline 2019-12-10T02:03:37.107874+0000) 2019-12-10T02:04:10.669 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:10.663+0000 1694f700 -1 osd.5 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.354634+0000 front 2019-12-10T02:02:49.330234+0000 (oldest deadline 2019-12-10T02:03:32.970354+0000) 2019-12-10T02:04:10.711 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:10.706+0000 1694f700 -1 osd.7 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:48.585820+0000 front 2019-12-10T02:02:48.314173+0000 (oldest deadline 2019-12-10T02:03:34.129684+0000) 2019-12-10T02:04:11.139 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:11.136+0000 1694f700 -1 osd.0 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.189172+0000 front 2019-12-10T02:02:51.156109+0000 (oldest deadline 2019-12-10T02:03:32.823640+0000) 2019-12-10T02:04:11.197 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:04:11.193+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 47sec e226 v226) 2019-12-10T02:04:11.245 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:04:11.238+0000 1694f700 -1 osd.4 226 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:02:51.302115+0000 front 2019-12-10T02:02:51.399113+0000 (oldest deadline 2019-12-10T02:03:34.163248+0000) 2019-12-10T02:04:11.568 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:11.514+0000 1694f700 -1 osd.6 227 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:12.686 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:12.583+0000 1694f700 -1 osd.6 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:13.611 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:04:13.608+0000 1694f700 -1 osd.6 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106647 2.32s0 2:4ef578c3:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_3736e2ef089e40e794ac014b7126dff9:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e226) 2019-12-10T02:04:51.581 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:51.452+0000 11caf700 -1 osd.0 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.653 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:51.641+0000 31f87700 -1 osd.5 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.665 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:51.648+0000 31f87700 -1 osd.5 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.670 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:51.461+0000 35f8f700 -1 osd.0 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.705 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:51.489+0000 32788700 -1 osd.0 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.730 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:51.718+0000 3578e700 -1 osd.7 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.816 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:51.513+0000 32788700 -1 osd.0 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.823 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:51.808+0000 31f87700 -1 osd.7 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.884 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:51.833+0000 35f8f700 -1 osd.5 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.929 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:51.906+0000 35f8f700 -1 osd.5 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.934 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:51.783+0000 32f89700 -1 osd.0 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:51.939 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:04:51.804+0000 3378a700 -1 osd.0 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:52.072 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:51.864+0000 36790700 -1 osd.7 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:52.081 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:52.013+0000 33f8b700 -1 osd.5 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:52.107 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:04:51.921+0000 3378a700 -1 osd.7 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:52.384 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:04:52.283+0000 36790700 -1 osd.5 230 build_incremental_map_msg missing incremental map 230 2019-12-10T02:04:54.939 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:04:54.766+0000 36f91700 -1 osd.3 232 build_incremental_map_msg missing incremental map 232 2019-12-10T02:04:55.060 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:04:54.841+0000 36f91700 -1 osd.3 232 build_incremental_map_msg missing incremental map 232 2019-12-10T02:04:55.068 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:04:54.880+0000 3478c700 -1 osd.3 232 build_incremental_map_msg missing incremental map 232 2019-12-10T02:04:55.079 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:04:54.910+0000 31f87700 -1 osd.3 232 build_incremental_map_msg missing incremental map 232 2019-12-10T02:04:56.745 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:04:56.725+0000 30f85700 -1 osd.3 232 build_incremental_map_msg missing incremental map 232 2019-12-10T02:05:09.296 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:09.284+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:09.582 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:05:09.574+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:09.021402+0000) 2019-12-10T02:05:09.902 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:09.899+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:10.342 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:10.336+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:10.610 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:05:10.604+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:09.021402+0000) 2019-12-10T02:05:10.620 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:10.615+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:10.946 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:10.943+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:11.129 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:11.122+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:11.135 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:11.130+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:11.372 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:11.366+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:11.599 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:05:11.594+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:09.021402+0000) 2019-12-10T02:05:11.661 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:11.648+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:11.935 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:11.931+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:12.100 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:12.093+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:12.191 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:12.184+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:12.373 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:12.366+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:12.585 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:05:12.580+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:09.021402+0000) 2019-12-10T02:05:12.645 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:12.638+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:12.958 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:12.952+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:13.141 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:13.135+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:13.149 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:13.146+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:13.420 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:13.414+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:13.599 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:05:13.588+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:09.021402+0000) 2019-12-10T02:05:13.659 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:13.650+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:13.944 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:13.942+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:14.136 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:14.134+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:14.187 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:14.180+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:14.479 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:14.464+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:14.691 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:14.680+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:14.966 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:14.962+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:15.169 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:15.166+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:15.170 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:15.168+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:15.225 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:15.218+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:15.479 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:15.473+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:15.720 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:15.714+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:15.991 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:15.989+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:16.161 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:16.155+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:16.161 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:16.157+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:16.226 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:16.220+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:16.517 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:16.513+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:16.759 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:16.752+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:16.947 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:16.944+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:17.209 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:17.204+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:17.209 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:17.207+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:17.266 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:17.259+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:17.504 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:17.497+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:17.717 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:17.709+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:17.916 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:17.913+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:18.192 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:18.188+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:18.193 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:18.190+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:18.284 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:18.273+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:18.544 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:18.539+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:18.745 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:18.739+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:18.876 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:18.872+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:19.216 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:19.211+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:19.216 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:19.214+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:19.328 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:19.321+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:19.514 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:19.508+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:19.764 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:19.757+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:19.839 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:19.834+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:20.170 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:20.167+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:20.170 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:20.168+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:20.374 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:20.368+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:20.532 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:20.525+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:20.745 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:20.739+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:20.804 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:20.800+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:21.205 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:21.201+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:21.206 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:21.203+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:21.431 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:21.422+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:21.573 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:21.567+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:21.762 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:21.756+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:21.790 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:21.785+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:22.158 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:22.156+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:22.160 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:22.158+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:22.403 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:22.397+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:22.548 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:22.541+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:22.743 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:22.736+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:22.793 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:22.789+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:23.193 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:23.184+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:23.193 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:23.188+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:23.359 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:23.356+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:23.425 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:23.416+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:23.540 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:23.531+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:23.775 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:23.768+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:23.788 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:23.784+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:24.187 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:24.183+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:24.189 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:24.187+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:24.394 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:24.388+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:24.488 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:24.466+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:24.570 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:24.563+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:24.747 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:24.744+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:24.782 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:24.776+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:25.229 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:25.226+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:25.230 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:25.228+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:25.403 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:25.397+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:25.433 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:25.423+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:25.593 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:25.577+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:25.765 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:25.760+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:25.773 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:25.766+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:26.272 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:26.268+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:26.272 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:26.269+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:26.388 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:26.383+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:26.409 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:26.406+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:26.560 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:26.552+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:26.745 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:26.741+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:26.779 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:26.773+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:27.229 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:27.225+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:27.229 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:27.227+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:27.417 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:27.410+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:27.428 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:27.424+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:27.521 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:27.515+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:27.772 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:27.766+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:27.789 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:27.785+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:28.221 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:28.218+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:28.222 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:28.220+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:28.421 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:28.408+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:28.427 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:28.424+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:28.564 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:28.558+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:28.781 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:28.775+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:28.790 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:28.783+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:29.263 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:29.259+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:29.263 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:29.261+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:29.389 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:29.382+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:29.415 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:29.412+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:29.536 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:29.529+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:29.749 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:29.740+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:29.768 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:29.764+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:30.270 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:30.264+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:30.270 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:30.267+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:30.369 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:30.365+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:30.371 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:30.368+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:30.393 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:30.387+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:30.588 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:30.582+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:30.734 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:30.728+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:30.749 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:30.742+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:31.321 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:31.316+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:31.517 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:31.319+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:31.518 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:31.355+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:31.518 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:31.391+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:31.518 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:31.394+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:31.618 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:31.611+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:31.701 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:31.697+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:31.796 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:31.789+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:32.277 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:32.273+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:32.277 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:32.275+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:32.399 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:32.395+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:32.405 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:32.401+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:32.413 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:32.406+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:32.661 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:32.654+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:32.696 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:32.691+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:32.795 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:32.789+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:33.294 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:33.289+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:33.294 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:33.292+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:33.395 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:33.392+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:33.399 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:33.397+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:33.423 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:33.416+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:33.627 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:33.620+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:33.737 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:33.723+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:33.835 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:33.827+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:34.335 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:34.329+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:34.338 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:34.335+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:34.400 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:34.393+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:34.443 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:34.438+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:34.444 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:34.441+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:34.641 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:34.635+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:34.713 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:34.710+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:34.885 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:34.879+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:35.349 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:35.344+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:35.349 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:35.347+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:35.380 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:35.374+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:35.462 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:35.460+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:35.465 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:35.463+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:35.637 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:35.618+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:35.732 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:35.727+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:35.934 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:35.928+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:36.368 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:36.361+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:36.392 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:36.386+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:36.394 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:36.391+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:36.493 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:36.488+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:36.494 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:36.490+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:36.586 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:36.579+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:36.691 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:36.687+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:36.969 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:36.960+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:37.347 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:37.339+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:37.431 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:37.428+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:37.432 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:37.429+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:37.485 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:37.479+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:37.486 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:37.483+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:37.606 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:37.597+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:37.718 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:37.715+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:37.945 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:37.938+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:38.326 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:38.315+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:38.434 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:38.430+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:38.437 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:38.434+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:38.491 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:38.486+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:38.492 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:38.489+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:38.615 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:38.609+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:38.713 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:38.711+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:38.916 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:38.910+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:39.302 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:39.295+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:39.465 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:39.458+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:39.465 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:39.462+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:39.487 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:39.483+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:39.487 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:39.485+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:39.583 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:39.576+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:39.760 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:39.757+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:39.922 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:39.914+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:40.278 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:40.268+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:40.458 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:40.454+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:40.461 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:40.456+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:40.507 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:40.503+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:40.508 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:40.505+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:40.567 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:40.560+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:40.790 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:40.787+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:40.956 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:40.950+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:41.245 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:41.238+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:41.408 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:05:41.404+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e232 v232) 2019-12-10T02:05:41.479 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:41.475+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:41.479 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:41.477+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:41.516 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:41.512+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:41.520 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:41.515+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:41.543 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:41.538+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:41.745 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:41.741+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:41.997 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:41.990+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:42.240 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:42.207+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:42.504 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:42.497+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:42.505 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:42.499+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:42.506 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:42.502+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:42.509 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:42.504+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:42.512 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:42.507+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:42.766 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:42.763+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:43.043 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:43.038+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:43.279 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:43.273+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:43.474 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:43.468+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:43.498 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:43.495+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:43.499 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:43.498+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:43.516 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:43.513+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:43.518 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:43.516+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:43.729 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:43.719+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:44.063 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:44.057+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:44.312 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:44.306+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:44.467 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:44.462+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:44.467 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:44.465+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:44.493 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:44.487+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:44.519 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:44.516+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:44.522 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:44.519+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:44.718 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:44.715+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:45.023 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:45.017+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:45.315 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:45.308+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:45.435 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:45.432+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:45.437 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:45.435+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:45.515 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:45.509+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:45.515 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:45.513+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:45.518 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:45.516+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:45.718 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:45.714+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:46.075 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:46.068+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:46.320 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:46.314+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:46.428 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:05:46.422+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e232 v232) 2019-12-10T02:05:46.468 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:46.465+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:46.469 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:46.467+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:46.486 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:46.480+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:46.487 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:46.483+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:46.488 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:46.486+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:46.749 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:46.746+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:47.071 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:47.064+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:47.323 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:47.311+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:47.458 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:47.455+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:47.459 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:47.458+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:47.490 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:47.485+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:47.519 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:47.515+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:47.521 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:47.518+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:47.753 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:47.750+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:48.109 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:48.104+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:48.355 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:48.349+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:48.459 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:48.456+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:48.460 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:48.458+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:48.491 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:48.486+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:48.492 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:48.490+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:48.512 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:48.505+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:48.716 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:48.713+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:49.146 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:49.139+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:49.359 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:49.353+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:49.463 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:49.459+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:49.466 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:49.464+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:49.524 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:49.521+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:49.526 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:49.523+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:49.529 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:49.523+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:49.758 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:49.757+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:49.916 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:32:10.979 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:05:49.925 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:32:10.988 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:05:49.929 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:32:10.993 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:05:49.933 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:32:10.996 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:05:49.951 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:32:11.110 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:05:49.962 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:32:11.121 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:05:50.150 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:50.144+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:50.173 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:32:11.237 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:05:50.269 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:32:11.324 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:05:50.288 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:32:11.445 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:05:50.377 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:50.367+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:50.505 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:50.502+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:50.505 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:50.504+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:50.519 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:50.512+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:50.568 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:50.564+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:50.569 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:50.568+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:50.724 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:50.721+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:51.140 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:51.134+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:51.423 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:51.416+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:51.486 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:51.483+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:51.487 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:51.485+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:51.504 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:05:51.502+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e232 v232) 2019-12-10T02:05:51.529 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:51.527+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:51.530 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:51.529+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:51.571 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:51.564+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:51.743 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:51.742+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:52.185 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:52.177+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:52.465 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:52.458+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:52.483 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:52.479+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:52.484 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:52.482+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:52.527 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:52.523+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:52.532 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:52.526+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:52.583 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:52.577+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:52.698 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:52.695+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:53.170 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:53.163+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:53.470 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:53.464+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:53.510 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:53.507+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:53.513 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:53.511+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:53.525 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:53.521+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:53.526 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:53.524+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:53.631 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:53.623+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:53.688 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:53.681+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:54.225 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:54.217+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:54.468 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:54.463+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:54.504 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:54.500+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:54.505 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:54.504+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:54.526 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:54.522+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:54.528 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:54.525+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:54.631 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:54.624+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:54.685 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:54.682+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:55.224 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:55.217+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:55.430 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:55.423+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:55.465 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:55.462+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:55.466 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:55.464+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:55.532 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:55.531+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:55.538 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:55.535+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:55.622 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:55.615+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:55.709 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:55.706+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:56.236 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:56.229+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:56.426 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:56.419+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:56.464 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:56.462+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:56.465 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:56.463+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:56.496 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:56.493+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:56.497 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:56.495+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:56.518 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:05:56.514+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e232 v232) 2019-12-10T02:05:56.611 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:56.605+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:56.697 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:56.684+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:57.219 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:57.212+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:57.393 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:57.384+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:57.441 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:57.439+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:57.444 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:57.443+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:57.464 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:57.462+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:57.466 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:57.465+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:57.661 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:57.655+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:57.669 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:57.666+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:58.202 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:58.196+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:58.407 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:58.400+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:58.450 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:58.445+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:58.451 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:58.449+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:58.470 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:58.466+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:58.470 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:58.469+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:58.614 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:58.609+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:58.633 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:58.630+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:58.757 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:05:58.752+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:58.428728+0000) 2019-12-10T02:05:59.183 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:05:59.176+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:05:59.418 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:05:59.412+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:05:59.481 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:59.477+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:05:59.481 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:05:59.480+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:05:59.483 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:59.478+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:05:59.484 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:05:59.481+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:05:59.601 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:05:59.594+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:05:59.672 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:05:59.671+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:05:59.737 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:05:59.733+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:58.428728+0000) 2019-12-10T02:06:00.215 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:06:00.209+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:06:00.460 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:06:00.454+0000 1694f700 -1 osd.7 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:26.994630+0000 front 2019-12-10T02:04:26.953878+0000 (oldest deadline 2019-12-10T02:05:11.021116+0000) 2019-12-10T02:06:00.479 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:00.476+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:06:00.492 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:00.486+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:06:00.515 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:00.513+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:06:00.516 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:00.516+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:06:00.600 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:06:00.594+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:06:00.686 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:06:00.682+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:06:00.698 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:06:00.692+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:58.428728+0000) 2019-12-10T02:06:01.179 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:06:01.173+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:06:01.501 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:01.496+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:06:01.502 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:01.500+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:06:01.505 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:01.500+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:06:01.506 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:01.503+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:06:01.526 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:06:01.524+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e232 v232) 2019-12-10T02:06:01.568 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:06:01.562+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:06:01.658 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:06:01.655+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:06:01.712 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:06:01.707+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:58.428728+0000) 2019-12-10T02:06:02.138 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:06:02.132+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:06:02.469 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:02.464+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:06:02.469 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:02.466+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:06:02.516 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:02.513+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:06:02.517 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:02.515+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:06:02.537 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:06:02.529+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:06:02.666 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:06:02.662+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:06:02.728 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:06:02.723+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:58.428728+0000) 2019-12-10T02:06:03.097 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:06:03.092+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:06:03.474 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:03.469+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:06:03.475 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:03.472+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:06:03.494 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:03.490+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:06:03.495 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:03.492+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:06:03.537 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:06:03.530+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:06:03.685 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:06:03.682+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:06:03.739 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:06:03.733+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:58.428728+0000) 2019-12-10T02:06:04.152 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:06:04.146+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:06:04.454 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:04.449+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:06:04.455 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:04.452+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:06:04.456 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:04.450+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:06:04.459 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:04.453+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:06:04.518 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:06:04.512+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:06:04.670 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:06:04.666+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:06:04.696 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:06:04.688+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:58.428728+0000) 2019-12-10T02:06:05.182 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:06:05.175+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:06:05.427 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:05.423+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:06:05.432 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:05.428+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:06:05.512 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:05.506+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:06:05.512 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:05.509+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:06:05.541 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:06:05.536+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:06:05.624 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:06:05.621+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:06:05.722 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:06:05.716+0000 1694f700 -1 osd.5 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.907856+0000 front 2019-12-10T02:04:24.114633+0000 (oldest deadline 2019-12-10T02:05:58.428728+0000) 2019-12-10T02:06:06.183 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:06:06.176+0000 1694f700 -1 osd.4 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:24.412604+0000 front 2019-12-10T02:04:24.458044+0000 (oldest deadline 2019-12-10T02:05:10.294023+0000) 2019-12-10T02:06:06.435 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:06.433+0000 1694f700 -1 osd.3 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:04:49.757287+0000 (oldest deadline 2019-12-10T02:05:29.757287+0000) 2019-12-10T02:06:06.437 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:06.435+0000 1694f700 -1 osd.3 232 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4557.0:106102 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e230) 2019-12-10T02:06:06.474 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:06.470+0000 1694f700 -1 osd.2 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:25.988657+0000 front 2019-12-10T02:04:26.071214+0000 (oldest deadline 2019-12-10T02:05:10.661450+0000) 2019-12-10T02:06:06.475 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:06:06.472+0000 1694f700 -1 osd.2 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106851 2.3as0 2.edd193ba (undecoded) ondisk+read+known_if_redirected e229) 2019-12-10T02:06:06.562 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:06:06.556+0000 1694f700 -1 osd.6 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:23.767209+0000 front 2019-12-10T02:04:23.334066+0000 (oldest deadline 2019-12-10T02:05:09.189610+0000) 2019-12-10T02:06:06.586 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:06:06.580+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e232 v232) 2019-12-10T02:06:06.673 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:06:06.669+0000 1694f700 -1 osd.0 232 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:04:28.023978+0000 front 2019-12-10T02:04:28.035139+0000 (oldest deadline 2019-12-10T02:05:09.109543+0000) 2019-12-10T02:06:07.461 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:06:07.432+0000 35f8f700 -1 osd.3 233 build_incremental_map_msg missing incremental map 232 2019-12-10T02:07:05.861 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:05.859+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:06.912 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:06.909+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:07.940 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:07.937+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:08.930 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:08.928+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:09.933 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:09.930+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:10.961 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:10.958+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:11.923 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:11.921+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:12.966 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:12.964+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:13.992 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:13.991+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:14.956 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:14.954+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:16.008 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:16.005+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:16.987 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:16.985+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:18.018 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:18.017+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:19.040 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:19.039+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:20.045 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:20.044+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:21.059 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:21.057+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:22.107 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:22.106+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:23.147 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:23.145+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:24.170 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:24.168+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:25.177 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:25.175+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:26.193 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:26.192+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:27.211 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:27.210+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:28.182 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:28.181+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:29.172 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:29.171+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:30.126 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:30.125+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:31.138 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:31.137+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:31.270 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:31.264+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:31.518 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:31.515+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:31.656 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:31.650+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:31.855 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:31.849+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:32.184 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:32.182+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:32.305 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:32.299+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:32.509 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:32.496+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:32.658 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:32.653+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:32.820 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:32.815+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:32.934 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:32.926+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:33.011 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:33.005+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:33.177 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:33.175+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:33.284 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:33.279+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:33.490 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:33.487+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:33.622 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:33.617+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:33.873 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:33.864+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:33.885 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:33.882+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:34.053 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:34.046+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:34.219 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:34.217+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:34.328 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:34.322+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:34.529 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:34.525+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:34.630 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:34.624+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:34.903 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:34.898+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:34.929 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:34.926+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:35.033 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:35.028+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:35.188 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:35.186+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:35.322 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:35.316+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:35.555 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:35.547+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:35.612 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:35.606+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:35.942 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:35.938+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:35.948 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:35.941+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:35.995 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:35.990+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:36.148 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:36.145+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:36.148 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:36.146+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:36.326 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:36.321+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:36.545 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:36.543+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:36.646 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:36.641+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:36.930 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:36.927+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:36.969 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:36.963+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:36.986 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:36.981+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:37.178 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:37.175+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:37.178 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:37.176+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:37.364 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:37.358+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:37.530 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:37.525+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:37.696 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:37.690+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:37.940 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:37.934+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:37.951 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:37.949+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:37.968 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:37.963+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:38.145 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:38.141+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:38.145 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:38.142+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:38.385 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:38.373+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:38.497 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:38.493+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:38.653 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:38.647+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:38.902 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:38.896+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:38.940 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:38.932+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:39.022 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:39.016+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:39.171 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:39.167+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:39.172 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:39.169+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:39.431 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:39.427+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:39.497 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:39.492+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:39.619 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:39.613+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:39.866 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:39.860+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:39.921 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:39.919+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:39.982 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:39.976+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:40.165 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:40.161+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:40.166 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:40.163+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:40.423 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:40.417+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:40.496 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:40.491+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:40.580 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:40.573+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:40.880 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:40.874+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:40.928 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:40.925+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:40.965 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:40.959+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:41.167 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:41.162+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:41.171 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:41.168+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:41.477 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:41.471+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:41.522 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:41.517+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:41.557 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:41.541+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:41.876 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:41.871+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:41.887 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:41.885+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:41.929 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:41.923+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:42.169 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:42.163+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:42.170 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:42.165+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:42.456 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:42.450+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:42.476 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:42.473+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:42.564 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:42.558+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:42.884 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:42.877+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:42.912 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:42.909+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:42.952 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:42.946+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:43.135 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:43.131+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:43.135 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:43.133+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:43.422 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:43.417+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:43.521 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:43.518+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:43.535 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:43.530+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:43.881 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:43.867+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:43.940 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:43.938+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:43.981 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:43.975+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:44.124 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:44.121+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:44.125 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:44.123+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:44.444 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:44.436+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:44.499 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:44.495+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:44.558 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:44.552+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:44.915 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:44.909+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:44.960 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:44.955+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:44.977 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:44.973+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:45.155 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:45.152+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:45.156 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:45.153+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:45.489 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:45.483+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:45.500 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:45.496+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:45.548 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:45.542+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:45.879 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:45.874+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:45.955 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:45.952+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:45.976 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:45.971+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:46.145 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:46.141+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:46.146 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:46.144+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:46.470 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:46.464+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:46.507 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:46.503+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:46.555 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:46.548+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:46.880 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:46.874+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:46.938 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:46.935+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:46.964 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:46.958+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:47.165 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:47.162+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:47.166 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:47.164+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:47.425 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:47.420+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:47.494 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:47.490+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:47.563 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:47.552+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:47.921 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:47.915+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:47.965 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:47.961+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:47.985 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:47.979+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:48.212 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:48.209+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:48.212 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:48.210+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:48.391 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:48.382+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:48.498 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:48.493+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:48.540 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:48.535+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:48.898 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:48.893+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:48.981 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:48.978+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:49.000 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:48.995+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:49.168 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:49.163+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:49.168 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:49.165+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:49.376 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:49.364+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:49.509 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:49.505+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:49.533 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:49.527+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:49.860 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:49.855+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:49.960 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:49.955+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:49.968 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:49.960+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:50.140 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:50.136+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:50.142 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:50.140+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:50.425 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:50.419+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:50.511 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:50.505+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:50.533 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:50.529+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:50.894 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:50.889+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:50.933 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:50.927+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:50.997 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:50.994+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:51.100 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:51.096+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:51.101 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:51.099+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:51.459 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:51.454+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:51.547 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:51.542+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:51.563 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:51.558+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:51.871 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:51.866+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:51.931 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:51.925+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:52.005 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:52.002+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:52.080 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:52.077+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:52.081 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:52.079+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:52.475 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:52.465+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:52.538 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:52.533+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:52.577 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:52.573+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:52.865 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:52.860+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:52.974 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:52.967+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:53.052 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:53.049+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:53.108 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:53.104+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:53.108 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:53.106+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:53.491 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:53.486+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:53.508 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:53.503+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:53.594 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:53.590+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:53.830 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:53.823+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:54.010 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:54.005+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:54.020 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:54.017+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:54.112 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:54.108+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:54.112 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:54.110+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:54.481 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:54.476+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:54.505 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:54.499+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:54.610 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:54.607+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:54.824 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:54.819+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:54.976 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:54.971+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:55.024 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:55.020+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:55.161 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:55.157+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:55.162 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:55.159+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:55.494 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:55.488+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:55.537 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:55.531+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:55.639 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:55.635+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:55.808 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:55.802+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:55.996 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:55.990+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:56.012 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:56.007+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:56.147 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:56.141+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:56.147 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:56.143+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:56.487 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:56.482+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:56.516 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:56.508+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:56.640 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:56.636+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:56.802 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:56.796+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:57.006 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:57.003+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:57.036 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:57.030+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:57.156 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:57.153+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:57.157 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:57.155+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:57.502 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:57.496+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:57.540 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:57.534+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:57.626 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:57.622+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:57.770 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:57.764+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:57.988 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:57.984+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:58.075 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:58.069+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:58.142 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:58.137+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:58.142 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:58.140+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:58.468 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:58.463+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:58.584 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:58.578+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:58.678 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:58.673+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:58.726 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:58.720+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:07:58.983 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:07:58.980+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:07:59.113 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:07:59.107+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:07:59.146 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:59.142+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:07:59.148 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:07:59.146+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:07:59.466 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:07:59.459+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:07:59.581 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:07:59.576+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:07:59.690 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:07:59.686+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:07:59.706 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:07:59.700+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:00.035 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:00.030+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:00.123 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:00.106+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:00.156 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:00.149+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:00.158 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:00.152+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:00.496 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:00.490+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:00.634 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:00.628+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:00.664 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:00.662+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:00.749 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:00.744+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:01.053 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:01.050+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:01.082 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:01.077+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:01.146 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:01.142+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:01.147 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:01.144+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:01.492 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:01.487+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:01.677 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:01.671+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:01.700 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:01.696+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:01.801 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:01.797+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:01.996 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:08:01.985+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 43sec e237 v237) 2019-12-10T02:08:02.012 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:02.010+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:02.086 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:02.080+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:02.111 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:02.110+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:02.113 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:02.112+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:02.510 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:02.505+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:02.703 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:02.697+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:02.765 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:02.750+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:02.792 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:02.788+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:02.973 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:02.968+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:03.105 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:03.100+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:03.108 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:03.106+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:03.110 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:03.110+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:03.517 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:03.512+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:03.708 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:03.703+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:03.742 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:03.739+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:03.751 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:03.745+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:03.974 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:03.971+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:04.118 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:04.112+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:04.126 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:04.123+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:04.126 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:04.125+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:04.550 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:04.541+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:04.674 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:04.668+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:04.752 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:04.746+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:04.763 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:04.761+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:05.016 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:05.014+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:05.115 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:05.113+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:05.115 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:05.114+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:05.172 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:05.167+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:05.575 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:05.569+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:05.640 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:05.634+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:05.715 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:05.710+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:05.776 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:05.773+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:06.057 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:06.055+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:06.133 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:06.129+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:06.133 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:06.132+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:06.156 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:06.151+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:06.610 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:06.605+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:06.628 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:06.623+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:06.726 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:06.721+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:06.768 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:06.764+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:07.001 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:08:06.998+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 43sec e237 v237) 2019-12-10T02:08:07.034 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:07.031+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:07.108 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:07.107+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:07.109 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:07.108+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:07.162 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:07.156+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:07.596 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:07.590+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:07.654 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:07.641+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:07.723 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:07.717+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:07.794 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:07.792+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:08.067 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:08.065+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:08.068 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:08.067+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:08.088 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:08.085+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:08.121 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:08.116+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:08.610 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:08.604+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:08.644 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:08.640+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:08.771 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:08.765+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:08.813 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:08.808+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:09.101 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:09.097+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:09.114 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:09.111+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:09.117 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:09.115+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:09.152 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:09.146+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:09.568 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:09.561+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:09.607 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:09.599+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:09.803 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:09.799+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:09.804 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:09.798+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:10.108 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:10.106+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:10.110 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:10.109+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:10.137 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:10.136+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:10.167 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:10.161+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:10.585 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:10.579+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:10.627 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:10.621+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:10.790 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:10.788+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:10.852 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:10.847+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:11.111 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:11.109+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:11.112 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:11.110+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:11.126 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:11.120+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:11.163 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:11.161+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:11.541 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:08:11.535+0000 1694f700 -1 osd.7 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.573051+0000 front 2019-12-10T02:06:49.559136+0000 (oldest deadline 2019-12-10T02:07:30.305312+0000) 2019-12-10T02:08:11.646 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:11.640+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:11.835 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:11.829+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:11.842 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:08:11.840+0000 1694f700 -1 osd.0 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:47.106692+0000 front 2019-12-10T02:06:47.183847+0000 (oldest deadline 2019-12-10T02:07:31.152627+0000) 2019-12-10T02:08:12.074 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:08:12.070+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 43sec e237 v237) 2019-12-10T02:08:12.096 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:12.095+0000 1694f700 -1 osd.2 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.861618+0000 front 2019-12-10T02:06:50.181793+0000 (oldest deadline 2019-12-10T02:07:35.746891+0000) 2019-12-10T02:08:12.099 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:12.097+0000 1694f700 -1 osd.2 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:12.136 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:08:12.134+0000 1694f700 -1 osd.1 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:49.560736+0000 front 2019-12-10T02:06:49.217664+0000 (oldest deadline 2019-12-10T02:07:32.631703+0000) 2019-12-10T02:08:12.153 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:08:12.138+0000 1694f700 -1 osd.6 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.369201+0000 front 2019-12-10T02:06:48.371087+0000 (oldest deadline 2019-12-10T02:07:32.517669+0000) 2019-12-10T02:08:12.639 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:08:12.633+0000 1694f700 -1 osd.5 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.321076+0000 front 2019-12-10T02:06:48.293948+0000 (oldest deadline 2019-12-10T02:07:31.096195+0000) 2019-12-10T02:08:12.821 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:12.817+0000 1694f700 -1 osd.4 237 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:06:48.491173+0000 front 2019-12-10T02:06:48.475913+0000 (oldest deadline 2019-12-10T02:07:31.151566+0000) 2019-12-10T02:08:13.112 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:13.109+0000 1694f700 -1 osd.2 238 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:14.075 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:14.074+0000 1694f700 -1 osd.2 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:15.122 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:15.120+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:16.123 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:16.122+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:17.150 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:17.149+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:18.132 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:18.129+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:19.177 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:19.175+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:20.203 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:20.200+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:21.158 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:21.155+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:22.174 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:22.172+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:23.132 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:23.129+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:24.100 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:24.098+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:25.098 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:25.095+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:26.081 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:26.079+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:27.082 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:27.080+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:28.133 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:28.131+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:29.087 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:29.085+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:30.087 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:30.084+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:31.086 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:31.084+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:32.090 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:32.088+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:33.086 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:33.084+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:34.113 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:34.111+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:35.127 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:35.125+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:36.088 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:36.087+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:37.052 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:37.049+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:38.074 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:38.071+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:39.091 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:39.089+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:40.103 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:40.101+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:41.117 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:41.115+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:42.154 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:42.152+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:43.198 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:43.196+0000 1694f700 -1 osd.2 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:44.196 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:44.194+0000 1694f700 -1 osd.2 241 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:45.223 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:45.221+0000 1694f700 -1 osd.2 241 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:46.183 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:46.181+0000 1694f700 -1 osd.2 241 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:47.232 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:47.232+0000 1694f700 -1 osd.2 241 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:48.253 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:48.251+0000 1694f700 -1 osd.2 241 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:49.316 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:49.289+0000 36790700 -1 osd.4 242 build_incremental_map_msg missing incremental map 242 2019-12-10T02:08:49.347 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:49.339+0000 1694f700 -1 osd.2 242 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:49.425 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:49.390+0000 35f8f700 -1 osd.4 242 build_incremental_map_msg missing incremental map 242 2019-12-10T02:08:49.435 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:49.401+0000 36790700 -1 osd.4 242 build_incremental_map_msg missing incremental map 242 2019-12-10T02:08:49.552 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:49.497+0000 36f91700 -1 osd.4 242 build_incremental_map_msg missing incremental map 242 2019-12-10T02:08:49.597 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:49.535+0000 32f89700 -1 osd.4 242 build_incremental_map_msg missing incremental map 242 2019-12-10T02:08:49.729 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:49.567+0000 37f93700 -1 osd.4 242 build_incremental_map_msg missing incremental map 242 2019-12-10T02:08:49.745 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:08:49.580+0000 32f89700 -1 osd.4 242 build_incremental_map_msg missing incremental map 242 2019-12-10T02:08:50.395 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:50.389+0000 1694f700 -1 osd.2 243 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:51.417 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:51.412+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:52.450 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:52.444+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:53.454 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:53.452+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:54.452 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:54.451+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:55.441 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:55.439+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:56.482 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:56.482+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:57.444 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:57.443+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:58.402 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:58.402+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:08:59.403 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:08:59.402+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:00.452 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:00.451+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:01.498 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:01.496+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:02.487 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:02.486+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:03.541 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:03.537+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:04.513 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:04.507+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:05.565 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:05.561+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:06.549 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:06.546+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:07.587 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:07.581+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:08.608 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:08.606+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:09.567 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:09.565+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:10.406 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:10.401+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:10.547 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:10.543+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:10.548 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:10.545+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:11.099 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:11.093+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:11.440 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:11.433+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:11.503 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:11.500+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:11.507 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:11.503+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:12.093 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:12.089+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:12.461 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:12.457+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:12.462 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:12.459+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:12.474 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:12.469+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:13.128 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:13.124+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:13.196 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:13.190+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:13.419 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:13.415+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:13.420 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:13.417+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:13.449 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:13.435+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:13.964 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:13.957+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:14.158 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:14.154+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:14.206 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:14.200+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:14.401 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:14.395+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:14.458 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:14.455+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:14.459 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:14.456+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:14.627 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:14.620+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:14.955 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:14.950+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:15.157 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:15.154+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:15.212 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:15.205+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:15.393 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:15.387+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:15.495 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:15.492+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:15.496 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:15.493+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:15.673 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:15.666+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:15.971 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:15.964+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:16.119 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:16.115+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:16.206 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:16.199+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:16.380 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:16.373+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:16.504 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:16.501+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:16.505 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:16.502+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:16.705 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:16.698+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:16.987 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:16.981+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:17.126 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:17.123+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:17.241 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:17.235+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:17.391 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:17.385+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:17.547 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:17.543+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:17.547 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:17.545+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:17.706 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:17.700+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:18.019 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:18.012+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:18.132 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:18.129+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:18.215 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:18.208+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:18.404 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:18.397+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:18.528 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:18.525+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:18.529 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:18.527+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:18.718 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:18.711+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:19.017 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:19.012+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:19.167 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:19.164+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:19.193 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:19.186+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:19.360 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:19.353+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:19.570 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:19.565+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:19.570 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:19.568+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:19.728 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:19.721+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:20.017 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:20.011+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:20.173 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:20.169+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:20.228 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:20.222+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:20.404 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:20.397+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:20.577 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:20.573+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:20.577 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:20.575+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:20.730 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:20.723+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:20.992 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:20.986+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:21.133 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:21.129+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:21.214 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:21.207+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:21.452 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:21.447+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:21.554 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:21.550+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:21.557 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:21.552+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:21.699 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:21.693+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:22.031 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:22.022+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:22.119 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:22.115+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:22.229 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:22.222+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:22.437 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:22.431+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:22.545 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:22.540+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:22.569 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:22.566+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:22.570 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:22.568+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:22.682 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:22.678+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:23.040 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:23.034+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:23.140 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:23.136+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:23.219 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:23.213+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:23.402 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:23.390+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:23.519 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:23.512+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:23.563 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:23.552+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:23.564 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:23.554+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:23.649 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:23.644+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:24.097 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:24.086+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:24.121 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:24.116+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:24.267 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:24.261+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:24.434 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:24.428+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:24.479 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:24.477+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:24.510 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:24.506+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:24.511 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:24.508+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:24.686 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:24.676+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:25.074 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:25.069+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:25.154 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:25.151+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:25.240 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:25.234+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:25.431 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:25.424+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:25.445 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:25.441+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:25.560 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:25.557+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:25.562 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:25.560+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:25.733 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:25.726+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:26.089 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:26.083+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:26.135 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:26.130+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:26.256 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:26.250+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:26.413 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:26.407+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:26.483 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:26.479+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:26.529 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:26.525+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:26.531 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:26.526+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:26.770 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:26.764+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:27.125 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:27.118+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:27.140 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:27.135+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:27.270 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:27.263+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:27.451 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:27.445+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:27.473 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:27.469+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:27.473 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:27.471+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:27.520 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:27.518+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:27.522 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:27.520+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:27.526 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:35:48.589 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:09:27.538 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:35:48.599 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:09:27.542 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:35:48.605 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:09:27.545 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:35:48.609 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:09:27.556 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:35:48.715 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:09:27.576 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:35:48.735 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:09:27.780 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:35:48.843 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:09:27.790 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:27.785+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:27.822 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:35:48.885 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:09:27.843 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:35:49.002 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:09:28.139 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:28.136+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:28.161 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:28.155+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:28.238 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:28.232+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:28.484 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:28.479+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:28.509 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:28.506+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:28.512 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:28.509+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:28.522 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:28.518+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:28.523 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:28.520+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:28.811 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:28.807+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:29.162 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:29.158+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:29.210 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:29.200+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:29.256 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:29.250+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:29.501 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:29.494+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:29.534 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:29.531+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:29.534 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:29.532+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:29.551 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:29.548+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:29.552 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:29.549+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:29.769 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:29.762+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:30.119 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:30.113+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:30.164 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:30.158+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:30.217 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:30.210+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:30.476 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:30.470+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:30.513 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:30.509+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:30.515 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:30.513+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:30.541 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:30.537+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:30.544 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:30.541+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:30.773 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:30.767+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:31.087 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:31.084+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:31.128 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:31.122+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:31.202 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:31.195+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:31.468 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:31.462+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:31.499 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:31.493+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:31.500 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:31.497+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:31.533 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:31.514+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:31.533 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:31.515+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:31.753 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:31.747+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:32.069 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:32.066+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:32.090 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:32.084+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:32.255 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:32.248+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:32.477 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:32.471+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:32.500 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:32.498+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:32.502 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:32.501+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:32.517 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:32.511+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:32.517 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:32.513+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:32.794 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:32.788+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:33.119 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:33.116+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:33.127 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:33.121+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:33.276 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:33.270+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:33.457 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:33.451+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:33.465 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:33.461+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:33.467 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:33.464+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:33.541 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:33.539+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:33.542 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:33.542+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:33.765 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:33.759+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:34.107 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:34.100+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:34.147 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:34.143+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:34.309 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:34.303+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:34.425 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:34.422+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:34.427 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:34.424+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:34.434 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:34.428+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:34.510 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:34.507+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:34.512 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:34.512+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:34.759 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:34.753+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:35.143 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:35.137+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:35.157 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:35.155+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:35.312 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:35.305+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:35.395 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:35.391+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:35.396 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:35.394+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:35.401 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:35.395+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:35.484 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:35.481+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:35.488 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:35.485+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:35.783 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:35.776+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:36.130 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:36.123+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:36.183 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:36.181+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:36.290 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:36.282+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:36.405 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:36.400+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:36.416 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:36.409+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:36.417 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:36.414+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:36.452 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:36.448+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:36.452 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:36.451+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:36.839 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:36.821+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:37.167 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:37.160+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:37.179 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:37.177+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:37.326 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:37.320+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:37.365 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:37.358+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:37.421 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:37.417+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:37.422 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:37.420+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:37.483 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:37.481+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:37.484 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:37.483+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:37.800 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:37.793+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:38.209 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:38.203+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:38.215 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:38.212+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:38.317 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:38.311+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:38.393 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:38.386+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:38.408 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:38.405+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:38.412 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:38.411+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:38.467 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:38.464+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:38.468 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:38.466+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:38.790 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:38.784+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:39.177 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:39.171+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:39.240 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:39.236+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:39.301 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:39.295+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:39.410 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:39.408+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:39.414 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:39.410+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:39.428 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:39.427+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:39.429 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:39.428+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:39.443 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:39.438+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:39.762 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:39.757+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:40.197 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:40.191+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:40.199 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:40.196+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:40.272 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:40.266+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:40.437 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:40.434+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:40.437 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:40.436+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:40.448 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:40.442+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:40.453 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:40.452+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:40.456 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:40.455+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:40.792 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:40.784+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:41.189 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:41.186+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:41.222 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:41.215+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:41.311 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:41.305+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:41.417 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:41.411+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:41.452 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:41.449+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:41.452 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:41.451+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:41.487 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:41.484+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:41.488 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:41.486+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:41.750 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:41.743+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:42.187 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:42.181+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:42.237 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:42.235+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:42.342 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:42.335+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:42.383 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:42.377+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:42.460 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:42.456+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:42.462 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:42.460+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:42.527 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:42.525+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:42.534 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:42.532+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:42.780 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:42.774+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:43.165 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:43.160+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:43.215 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:43.211+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:43.393 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:43.386+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:43.396 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:43.389+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:43.422 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:43.421+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:43.423 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:43.423+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:43.512 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:43.509+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:43.512 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:43.511+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:43.833 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:43.828+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:44.144 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:44.138+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:44.266 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:44.264+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:44.377 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:44.368+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:44.401 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:44.398+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:44.402 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:44.400+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:44.419 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:44.413+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:44.469 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:44.465+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:44.471 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:44.468+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:44.860 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:44.854+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:45.152 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:45.145+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:45.246 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:45.244+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:45.332 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:45.325+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:45.378 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:45.372+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:45.422 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:45.419+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:45.422 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:45.421+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:45.520 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:45.515+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:45.520 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:45.519+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:45.858 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:45.853+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:46.152 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:46.146+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:46.257 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:46.254+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:46.337 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:46.332+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:46.358 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:46.353+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:46.385 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:46.382+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:46.386 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:46.386+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:46.498 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:46.494+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:46.504 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:46.501+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:46.884 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:46.878+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:47.162 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:47.156+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:47.276 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:47.272+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:47.388 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:47.382+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:47.406 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:47.400+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:47.413 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:47.409+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:47.416 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:47.414+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:47.444 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:09:47.440+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e244 v244) 2019-12-10T02:09:47.517 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:47.515+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:47.520 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:47.518+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:47.858 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:47.852+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:48.136 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:48.127+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:48.251 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:48.248+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:48.350 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:48.344+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:48.366 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:48.360+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:48.382 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:48.378+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:48.383 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:48.380+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:48.567 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:48.562+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:48.570 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:48.565+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:48.856 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:48.850+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:49.135 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:49.129+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:49.226 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:49.223+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:49.357 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:49.351+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:49.359 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:49.353+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:49.430 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:49.426+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:49.430 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:49.428+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:49.553 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:49.550+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:49.555 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:49.553+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:49.852 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:49.845+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:50.102 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:50.096+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:50.273 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:50.269+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:50.339 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:50.332+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:50.375 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:50.369+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:50.413 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:50.409+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:50.414 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:50.411+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:50.583 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:50.574+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:50.584 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:50.576+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:50.895 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:50.889+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:51.060 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:51.054+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:51.279 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:51.274+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:51.338 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:51.332+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:51.339 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:51.334+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:51.433 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:51.430+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:51.434 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:51.431+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:51.612 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:51.608+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:51.614 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:51.610+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:51.931 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:51.925+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:52.039 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:52.033+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:52.309 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:52.305+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:52.338 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:52.332+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:52.352 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:52.347+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:52.421 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:52.418+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:52.422 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:52.420+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:52.465 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:09:52.460+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e244 v244) 2019-12-10T02:09:52.575 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:52.572+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:52.578 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:52.576+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:52.957 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:52.951+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:53.077 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:53.071+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:53.298 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:53.294+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:53.327 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:53.321+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:53.367 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:53.362+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:53.410 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:53.406+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:53.411 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:53.409+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:53.614 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:53.608+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:53.616 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:53.612+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:54.008 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:54.002+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:54.052 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:54.044+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:54.317 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:54.313+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:54.377 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:54.370+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:54.380 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:54.374+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:54.435 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:54.432+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:54.437 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:54.435+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:54.609 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:54.607+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:54.612 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:54.609+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:55.059 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:55.051+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:55.080 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:55.073+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:55.332 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:55.324+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:55.351 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:55.347+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:55.401 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:55.395+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:55.489 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:55.486+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:55.492 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:55.490+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:55.630 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:55.628+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:55.633 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:55.631+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:56.035 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:56.028+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:56.038 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:56.032+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:56.321 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:56.312+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:56.369 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:56.366+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:56.449 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:56.444+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:56.450 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:56.447+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:56.452 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:56.446+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:56.615 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:56.611+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:56.616 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:56.613+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:57.004 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:56.999+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:57.027 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:57.021+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:57.314 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:57.309+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:57.382 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:57.379+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:57.437 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:57.429+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:57.447 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:57.442+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:57.447 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:57.444+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:57.476 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:09:57.474+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e244 v244) 2019-12-10T02:09:57.600 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:57.596+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:57.601 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:57.599+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:57.981 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:57.973+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:58.055 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:58.048+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:58.289 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:58.282+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:58.405 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:58.395+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:58.430 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:58.417+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:58.457 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:58.454+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:58.458 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:58.455+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:58.612 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:58.609+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:58.613 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:58.611+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:58.975 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:58.970+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:09:59.086 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:09:59.080+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:09:59.315 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:09:59.309+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:09:59.380 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:09:59.374+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:09:59.446 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:09:59.442+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:09:59.507 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:59.503+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:09:59.508 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:09:59.506+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:09:59.665 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:59.661+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:09:59.668 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:09:59.664+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:09:59.966 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:09:59.961+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:10:00.049 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:10:00.043+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:10:00.301 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:10:00.295+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:10:00.341 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:10:00.335+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:10:00.487 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:10:00.482+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:10:00.498 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:10:00.487+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:10:00.498 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:10:00.490+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:10:00.634 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:10:00.630+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:10:00.637 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:10:00.634+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:10:00.933 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:10:00.927+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:10:01.020 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:10:01.013+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:10:01.322 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:10:01.315+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:10:01.367 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:10:01.361+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:10:01.480 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:10:01.477+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:10:01.536 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:10:01.532+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:10:01.569 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:10:01.538+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:10:01.611 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:10:01.606+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:10:01.611 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:10:01.609+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:10:01.989 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:10:01.981+0000 1694f700 -1 osd.4 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.314096+0000 front 2019-12-10T02:08:28.238803+0000 (oldest deadline 2019-12-10T02:09:13.957049+0000) 2019-12-10T02:10:02.027 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:10:02.019+0000 1694f700 -1 osd.6 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.016394+0000 front 2019-12-10T02:08:29.045637+0000 (oldest deadline 2019-12-10T02:09:14.381470+0000) 2019-12-10T02:10:02.334 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:10:02.328+0000 1694f700 -1 osd.7 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.319463+0000 front 2019-12-10T02:08:29.230490+0000 (oldest deadline 2019-12-10T02:09:10.307113+0000) 2019-12-10T02:10:02.409 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:10:02.403+0000 1694f700 -1 osd.5 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.672010+0000 front 2019-12-10T02:08:29.530001+0000 (oldest deadline 2019-12-10T02:09:12.407838+0000) 2019-12-10T02:10:02.437 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:10:02.433+0000 1694f700 -1 osd.0 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:29.429862+0000 front 2019-12-10T02:08:29.484210+0000 (oldest deadline 2019-12-10T02:09:10.508265+0000) 2019-12-10T02:10:02.548 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:10:02.544+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 43sec e244 v244) 2019-12-10T02:10:02.572 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:10:02.568+0000 1694f700 -1 osd.2 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:08:28.660297+0000 front 2019-12-10T02:08:28.661345+0000 (oldest deadline 2019-12-10T02:09:10.172024+0000) 2019-12-10T02:10:02.575 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:10:02.573+0000 1694f700 -1 osd.2 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] ondisk+write+known_if_redirected e237) 2019-12-10T02:10:02.659 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:10:02.656+0000 1694f700 -1 osd.3 244 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:08:46.856305+0000 (oldest deadline 2019-12-10T02:09:26.856305+0000) 2019-12-10T02:10:02.663 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:10:02.661+0000 1694f700 -1 osd.3 244 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e242) 2019-12-10T02:11:01.877 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:01.875+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:02.868 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:02.866+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:03.858 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:03.856+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:04.880 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:04.879+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:05.896 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:05.894+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:06.847 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:06.846+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:07.869 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:07.868+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:08.864 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:08.862+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:09.905 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:09.903+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:10.864 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:10.862+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:11.915 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:11.913+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:12.904 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:12.902+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:13.905 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:13.904+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:14.944 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:14.943+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:15.996 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:15.994+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:16.991 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:16.988+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:18.002 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:17.999+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:19.006 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:19.003+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:20.034 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:20.031+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:20.998 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:20.996+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:22.004 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:22.002+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:23.042 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:23.039+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:23.201 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:37:44.264 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:11:23.204 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:37:44.268 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:11:23.234 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:37:44.297 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:11:23.249 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:37:44.312 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:11:23.277 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:37:44.436 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:11:23.282 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:37:44.441 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:11:23.387 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:37:44.450 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:11:23.389 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:37:44.452 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:11:23.425 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:37:44.584 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:11:23.999 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:23.997+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:24.983 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:24.981+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:25.941 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:25.939+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:26.988 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:26.986+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:27.990 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:27.988+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:28.975 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:28.973+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:29.937 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:29.932+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:30.974 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:30.972+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:31.932 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:31.930+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:32.613 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:32.606+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:32.911 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:32.909+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:33.248 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:33.242+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:33.611 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:33.605+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:33.904 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:33.902+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:34.259 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:34.253+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:34.571 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:34.566+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:34.642 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:34.636+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:34.895 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:34.892+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:34.896 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:34.893+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:35.251 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:35.245+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:35.278 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:35.275+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:35.559 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:35.547+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:35.694 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:35.687+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:35.845 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:35.841+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:35.881 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:35.877+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:35.882 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:35.879+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:36.271 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:36.265+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:36.303 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:36.298+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:36.522 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:36.515+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:36.734 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:36.728+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:36.817 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:36.811+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:36.872 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:36.867+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:36.919 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:36.916+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:36.920 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:36.918+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:37.264 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:37.253+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:37.292 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:37.288+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:37.493 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:37.486+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:37.705 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:37.699+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:37.867 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:37.860+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:37.876 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:37.872+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:37.938 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:37.934+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:37.938 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:37.936+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:38.274 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:38.269+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:38.316 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:38.314+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:38.492 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:38.486+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:38.716 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:38.708+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:38.846 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:38.842+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:38.857 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:38.852+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:38.942 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:38.939+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:38.945 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:38.940+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:39.228 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:39.223+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:39.285 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:39.282+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:39.500 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:39.495+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:39.722 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:39.717+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:39.833 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:39.830+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:39.899 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:39.893+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:39.980 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:39.976+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:39.980 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:39.978+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:40.239 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:40.234+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:40.244 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:40.240+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:40.517 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:40.511+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:40.747 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:40.742+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:40.867 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:40.864+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:40.876 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:40.871+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:40.994 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:40.992+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:40.995 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:40.993+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:41.244 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:41.241+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:41.293 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:41.287+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:41.518 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:41.513+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:41.740 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:41.734+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:41.849 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:41.845+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:41.861 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:41.855+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:41.981 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:41.978+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:41.982 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:41.979+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:42.220 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:42.217+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:42.339 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:42.333+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:42.480 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:42.474+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:42.717 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:42.711+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:42.822 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:42.818+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:42.903 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:42.896+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:43.024 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:43.019+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:43.025 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:43.020+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:43.187 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:43.184+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:43.320 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:43.314+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:43.438 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:43.433+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:43.674 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:43.669+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:43.846 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:43.843+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:43.957 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:43.951+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:44.034 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:44.030+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:44.035 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:44.032+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:44.193 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:44.189+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:44.372 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:44.367+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:44.490 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:44.484+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:44.658 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:44.652+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:44.870 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:44.866+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:44.950 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:44.944+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:45.011 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:45.007+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:45.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:45.010+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:45.217 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:45.213+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:45.382 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:45.377+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:45.492 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:45.487+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:45.635 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:45.629+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:45.886 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:45.882+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:45.967 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:45.961+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:45.987 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:45.983+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:45.988 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:45.985+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:46.178 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:46.173+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:46.428 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:46.423+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:46.512 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:46.506+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:46.666 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:46.660+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:46.915 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:46.911+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:46.939 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:46.934+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:47.000 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:46.997+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:47.001 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:46.999+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:47.185 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:47.182+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:47.419 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:47.414+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:47.532 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:47.526+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:47.683 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:47.677+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:47.874 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:47.871+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:47.964 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:47.958+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:47.969 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:47.968+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:47.970 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:47.969+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:48.150 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:48.147+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:48.388 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:48.383+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:48.558 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:48.554+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:48.709 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:48.703+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:48.897 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:48.895+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:48.974 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:48.946+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:48.975 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:48.969+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:48.975 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:48.971+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:49.127 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:49.124+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:49.438 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:49.427+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:49.615 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:49.609+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:49.691 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:49.686+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:49.888 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:49.885+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:49.932 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:49.927+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:50.018 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:50.017+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:50.019 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:50.018+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:50.083 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:50.081+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:50.484 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:50.478+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:50.615 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:50.609+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:50.716 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:50.710+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:50.895 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:50.893+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:50.898 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:50.893+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:51.025 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:51.023+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:51.026 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:51.025+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:51.043 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:51.040+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:51.469 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:51.463+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:51.598 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:51.591+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:51.687 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:51.683+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:51.947 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:51.945+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:51.952 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:51.948+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:52.018 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:52.015+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:52.019 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:52.017+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:52.019 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:52.017+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:52.491 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:52.486+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:52.557 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:52.549+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:52.688 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:52.683+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:52.978 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:52.976+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:52.979 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:52.977+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:52.989 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:52.987+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:53.002 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:52.997+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:53.057 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:53.053+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:53.505 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:53.500+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:53.540 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:53.534+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:53.700 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:53.695+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:53.950 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:53.947+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:53.951 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:53.949+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:53.995 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:53.991+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:54.021 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:54.019+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:54.024 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:54.019+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:54.461 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:54.455+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:54.588 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:54.582+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:54.671 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:54.666+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:54.982 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:54.980+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:54.998 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:54.997+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:54.999 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:54.998+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:55.045 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:55.036+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:55.074 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:55.072+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:55.500 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:55.494+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:55.635 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:55.630+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:55.678 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:55.672+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:55.937 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:55.935+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:55.969 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:55.965+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:55.970 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:55.968+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:56.052 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:56.048+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:56.058 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:56.052+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:56.496 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:56.491+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:56.669 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:56.662+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:56.687 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:56.681+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:56.914 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:56.912+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:57.004 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:57.000+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:57.005 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:57.003+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:57.071 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:57.065+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:57.097 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:57.094+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:57.491 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:57.485+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:57.652 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:57.646+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:57.653 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:57.647+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:57.955 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:57.952+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:57.984 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:57.981+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:57.984 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:57.983+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:58.041 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:58.035+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:58.107 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:58.105+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:58.547 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:58.541+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:58.668 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:58.662+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:58.702 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:58.696+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:58.957 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:58.954+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:11:59.003 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:59.001+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:11:59.004 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:11:59.003+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:11:59.012 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:11:59.006+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:11:59.068 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:11:59.067+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:11:59.559 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:11:59.553+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:11:59.701 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:11:59.695+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:11:59.731 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:11:59.726+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:11:59.982 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:11:59.978+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:00.008 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:00.001+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:00.035 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:00.032+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:00.035 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:00.034+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:00.059 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:00.053+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:00.556 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:00.550+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:00.737 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:00.731+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:00.741 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:00.735+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:00.969 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:00.964+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:01.015 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:01.013+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:01.027 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:01.025+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:01.028 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:01.026+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:01.070 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:01.068+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:01.581 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:01.575+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:01.744 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:01.738+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:01.786 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:01.780+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:01.923 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:01.918+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:02.013 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:02.008+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:02.013 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:02.010+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:02.066 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:02.063+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:02.106 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:02.103+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:02.634 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:02.623+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:02.739 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:02.734+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:02.782 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:02.776+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:02.961 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:02.955+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:03.030 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:03.026+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:03.031 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:03.028+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:03.050 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:03.047+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:03.144 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:03.141+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:03.258 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:12:03.253+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 43sec e249 v249) 2019-12-10T02:12:03.683 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:03.673+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:03.704 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:03.698+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:03.807 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:03.801+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:03.960 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:03.955+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:04.051 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:04.048+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:04.074 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:04.070+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:04.074 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:04.072+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:04.105 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:04.101+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:04.730 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:04.725+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:04.733 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:04.726+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:04.832 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:04.827+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:04.941 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:04.936+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:05.008 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:05.005+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:05.033 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:05.029+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:05.033 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:05.031+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:05.093 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:05.090+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:05.711 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:05.698+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:05.783 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:05.778+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:05.827 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:05.821+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:05.939 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:05.933+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:06.049 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:06.046+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:06.059 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:06.053+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:06.060 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:06.057+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:06.143 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:06.140+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:06.663 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:06.657+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:06.797 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:06.791+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:06.806 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:06.801+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:06.986 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:06.980+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:07.022 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:07.018+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:07.024 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:07.020+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:07.024 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:07.022+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:07.137 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:07.130+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:07.643 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:07.634+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:07.798 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:07.792+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:07.853 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:07.845+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:07.977 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:07.974+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:07.977 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:07.975+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:07.997 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:07.992+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:08.029 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:08.026+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:08.177 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:08.170+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:08.273 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:12:08.267+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 43sec e249 v249) 2019-12-10T02:12:08.669 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:08.663+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:08.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:08.802+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:08.888 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:08.882+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:08.952 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:08.946+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:08.952 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:08.948+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:09.025 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:09.019+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:09.054 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:09.051+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:09.170 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:09.167+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:09.677 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:09.671+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:09.788 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:09.782+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:09.932 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:09.927+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:09.963 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:09.961+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:09.964 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:09.962+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:10.007 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:10.001+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:10.054 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:10.051+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:10.189 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:10.187+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:10.671 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:10.666+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:10.788 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:10.783+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:10.948 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:10.944+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:10.948 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:10.945+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:10.973 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:10.968+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:10.991 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:10.983+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:11.010 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:11.008+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:11.162 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:11.158+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:11.674 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:11.668+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:11.762 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:11.757+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:11.915 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:11.913+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:11.916 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:11.914+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:11.979 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:11.972+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:12.000 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:11.994+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:12.048 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:12.044+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:12.153 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:12.150+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:12.698 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:12:12.693+0000 1694f700 -1 osd.7 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.197853+0000 front 2019-12-10T02:10:49.566311+0000 (oldest deadline 2019-12-10T02:11:32.045380+0000) 2019-12-10T02:12:12.765 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:12:12.760+0000 1694f700 -1 osd.5 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.973876+0000 front 2019-12-10T02:10:49.983263+0000 (oldest deadline 2019-12-10T02:11:32.842353+0000) 2019-12-10T02:12:12.886 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:12.882+0000 1694f700 -1 osd.2 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:52.074447+0000 front 2019-12-10T02:10:52.075044+0000 (oldest deadline 2019-12-10T02:11:34.303572+0000) 2019-12-10T02:12:12.887 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:12.884+0000 1694f700 -1 osd.2 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:12.933 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:12:12.927+0000 1694f700 -1 osd.6 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.122700+0000 front 2019-12-10T02:10:51.145380+0000 (oldest deadline 2019-12-10T02:11:36.198538+0000) 2019-12-10T02:12:13.008 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:12:13.002+0000 1694f700 -1 osd.4 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.966950+0000 front 2019-12-10T02:10:49.957618+0000 (oldest deadline 2019-12-10T02:11:33.990811+0000) 2019-12-10T02:12:13.053 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:12:13.051+0000 1694f700 -1 osd.0 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:51.810377+0000 front 2019-12-10T02:10:51.811157+0000 (oldest deadline 2019-12-10T02:11:35.148336+0000) 2019-12-10T02:12:13.191 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:12:13.189+0000 1694f700 -1 osd.1 249 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:10:49.851191+0000 front 2019-12-10T02:10:49.858129+0000 (oldest deadline 2019-12-10T02:11:34.317664+0000) 2019-12-10T02:12:13.323 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:12:13.320+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 43sec e249 v249) 2019-12-10T02:12:14.104 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:14.064+0000 1694f700 -1 osd.2 250 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:15.149 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:15.133+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:16.092 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:16.090+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:17.134 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:17.131+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:18.122 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:18.120+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:19.158 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:19.155+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:20.194 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:20.192+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:21.221 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:21.217+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:22.223 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:22.221+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:23.188 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:23.186+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:24.244 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:24.236+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:25.210 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:25.206+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:26.201 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:26.198+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:27.199 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:27.197+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:28.202 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:28.200+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:29.173 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:29.171+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:30.200 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:30.197+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:31.249 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:31.247+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:32.273 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:32.271+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:33.250 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:33.246+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:34.267 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:34.266+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:35.293 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:35.292+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:36.249 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:36.247+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:37.261 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:37.260+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:38.242 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:38.241+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:39.235 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:39.234+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:40.190 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:40.189+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:41.206 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:41.205+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:42.256 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:42.255+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:43.247 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:43.245+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:44.261 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:44.260+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:45.284 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:45.282+0000 1694f700 -1 osd.2 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:46.243 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:46.243+0000 1694f700 -1 osd.2 252 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:47.260 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:47.259+0000 1694f700 -1 osd.2 252 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:48.198 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:12:48.128+0000 1694f700 -1 osd.3 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:106116 7.2 7:45e713bc:::2019-12-10-02-1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.670-6fe178bb50aa4bcaa725ba04c611f6dc:head [append 0~489 in=489b] snapc 0=[] RETRY=8 ondisk+retry+write+known_if_redirected e248) 2019-12-10T02:12:48.274 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:48.271+0000 1694f700 -1 osd.2 252 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:49.247 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:49.245+0000 1694f700 -1 osd.2 252 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:50.228 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:50.226+0000 1694f700 -1 osd.2 252 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:51.266 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:51.264+0000 1694f700 -1 osd.2 253 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:52.283 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:52.280+0000 1694f700 -1 osd.2 254 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:53.264 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:12:53.245+0000 31786700 -1 osd.3 254 build_incremental_map_msg missing incremental map 254 2019-12-10T02:12:53.300 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:53.297+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:53.340 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:12:53.281+0000 31786700 -1 osd.3 254 build_incremental_map_msg missing incremental map 254 2019-12-10T02:12:53.372 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:12:53.339+0000 37f93700 -1 osd.3 254 build_incremental_map_msg missing incremental map 254 2019-12-10T02:12:53.581 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:12:53.409+0000 34f8d700 -1 osd.3 254 build_incremental_map_msg missing incremental map 254 2019-12-10T02:12:53.593 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:12:53.428+0000 31f87700 -1 osd.3 254 build_incremental_map_msg missing incremental map 254 2019-12-10T02:12:53.605 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:12:53.461+0000 3578e700 -1 osd.3 254 build_incremental_map_msg missing incremental map 254 2019-12-10T02:12:53.745 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:12:53.627+0000 31f87700 -1 osd.3 254 build_incremental_map_msg missing incremental map 254 2019-12-10T02:12:54.348 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:54.346+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:55.335 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:55.333+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:56.342 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:56.341+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:57.365 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:57.364+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:58.384 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:58.382+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:12:59.359 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:12:59.357+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:00.377 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:00.375+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:01.348 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:01.346+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:02.373 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:02.371+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:03.387 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:03.386+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:04.400 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:04.398+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:05.388 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:05.386+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:06.348 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:06.345+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:07.376 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:07.373+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:08.389 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:08.387+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:09.037 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:39:30.100 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:13:09.045 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:39:30.108 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:13:09.054 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:39:30.117 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:13:09.060 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:39:30.123 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:13:09.060 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:39:30.220 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:13:09.067 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:39:30.226 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:13:09.227 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:39:30.381 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:13:09.243 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:39:30.306 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:13:09.256 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:39:30.319 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:13:09.366 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:09.364+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:10.352 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:10.350+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:11.380 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:11.378+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:11.638 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:11.632+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:12.049 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:12.043+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:12.410 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:12.408+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:12.478 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:12.473+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:12.617 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:12.611+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:13.027 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:13.016+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:13.076 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:13.067+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:13.414 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:13.410+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:13.448 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:13.441+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:13.656 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:13.651+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:14.055 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:14.050+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:14.110 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:14.102+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:14.399 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:14.397+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:14.494 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:14.488+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:14.690 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:14.685+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:15.056 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:15.050+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:15.160 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:15.154+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:15.363 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:15.359+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:15.445 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:15.442+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:15.447 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:15.445+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:15.501 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:15.494+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:15.702 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:15.695+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:16.045 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:16.038+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:16.153 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:16.148+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:16.399 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:16.393+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:16.409 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:16.406+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:16.410 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:16.407+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:16.546 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:16.540+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:16.732 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:16.726+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:17.023 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:17.017+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:17.200 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:17.193+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:17.366 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:17.362+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:17.366 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:17.365+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:17.390 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:17.387+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:17.587 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:17.581+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:17.691 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:17.685+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:18.068 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:18.062+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:18.194 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:18.187+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:18.399 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:18.396+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:18.399 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:18.398+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:18.423 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:18.422+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:18.570 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:18.561+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:18.706 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:18.699+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:19.096 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:19.088+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:19.197 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:19.189+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:19.364 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:19.361+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:19.365 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:19.363+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:19.427 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:19.423+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:19.611 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:19.603+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:19.730 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:19.723+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:20.133 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:20.125+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:20.166 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:20.158+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:20.348 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:20.346+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:20.350 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:20.349+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:20.447 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:20.445+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:20.590 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:20.584+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:20.711 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:20.704+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:21.122 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:21.116+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:21.185 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:21.172+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:21.337 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:21.336+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:21.339 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:21.339+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:21.400 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:21.398+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:21.600 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:21.594+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:21.764 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:21.758+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:22.134 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:22.129+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:22.158 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:22.153+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:22.300 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:22.296+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:22.300 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:22.298+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:22.422 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:22.416+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:22.641 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:22.635+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:22.734 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:22.728+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:23.152 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:23.147+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:23.209 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:23.202+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:23.283 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:23.279+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:23.284 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:23.282+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:23.414 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:23.411+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:23.475 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:23.473+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:23.682 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:23.673+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:23.737 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:23.731+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:24.129 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:24.123+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:24.186 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:24.178+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:24.285 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:24.282+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:24.287 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:24.286+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:24.390 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:24.386+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:24.460 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:24.459+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:24.648 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:24.643+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:24.747 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:24.740+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:25.104 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:25.099+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:25.217 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:25.211+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:25.308 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:25.305+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:25.309 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:25.307+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:25.350 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:25.348+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:25.496 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:25.493+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:25.680 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:25.674+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:25.792 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:25.786+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:26.067 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:26.061+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:26.262 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:26.255+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:26.275 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:26.273+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:26.276 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:26.274+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:26.348 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:26.347+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:26.491 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:26.489+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:26.733 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:26.727+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:26.786 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:26.779+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:27.077 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:27.072+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:27.253 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:27.250+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:27.254 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:27.253+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:27.270 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:27.263+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:27.356 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:27.354+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:27.482 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:27.480+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:27.745 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:27.738+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:27.760 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:27.754+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:28.038 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:28.032+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:28.247 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:28.244+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:28.248 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:28.246+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:28.311 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:28.303+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:28.312 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:28.309+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:28.533 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:28.530+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:28.539 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:28.533+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:28.750 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:28.743+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:28.799 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:28.793+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:29.007 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:29.001+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:29.293 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:29.290+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:29.293 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:29.292+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:29.315 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:29.309+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:29.346 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:29.342+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:29.563 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:29.560+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:29.567 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:29.565+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:29.768 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:29.763+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:29.822 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:29.811+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:30.006 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:30.000+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:30.266 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:30.263+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:30.268 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:30.267+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:30.326 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:30.324+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:30.356 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:30.351+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:30.600 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:30.597+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:30.601 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:30.599+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:30.805 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:30.799+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:30.821 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:30.815+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:30.977 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:30.970+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:31.258 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:31.255+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:31.259 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:31.257+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:31.291 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:31.287+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:31.316 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:31.311+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:31.556 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:31.553+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:31.559 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:31.558+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:31.780 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:31.774+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:31.781 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:31.775+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:32.024 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:32.018+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:32.248 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:32.232+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:32.248 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:32.235+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:32.286 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:32.279+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:32.322 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:32.319+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:32.524 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:32.519+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:32.526 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:32.522+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:32.764 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:32.758+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:32.817 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:32.812+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:33.013 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:33.007+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:33.227 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:33.222+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:33.235 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:33.232+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:33.288 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:33.283+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:33.300 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:33.298+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:33.572 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:33.566+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:33.572 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:33.568+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:33.786 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:33.779+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:33.792 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:33.786+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:34.041 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:34.036+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:34.269 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:34.262+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:34.281 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:34.276+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:34.281 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:34.278+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:34.316 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:34.311+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:34.603 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:34.600+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:34.604 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:34.602+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:34.792 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:34.786+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:34.828 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:34.823+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:34.999 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:34.992+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:35.249 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:35.243+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:35.250 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:35.246+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:35.253 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:35.251+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:35.333 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:35.330+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:35.656 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:35.651+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:35.659 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:35.654+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:35.786 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:35.780+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:35.829 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:35.820+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:35.961 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:35.953+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:36.218 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:36.213+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:36.218 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:36.216+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:36.281 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:36.274+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:36.354 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:36.349+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:36.681 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:36.677+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:36.685 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:36.682+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:36.745 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:36.738+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:36.798 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:36.792+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:36.949 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:36.943+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:37.173 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:37.169+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:37.175 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:37.173+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:37.333 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:37.327+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:37.360 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:37.355+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:37.713 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:37.707+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:37.714 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:37.709+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:37.724 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:37.718+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:37.841 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:37.834+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:38.001 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:37.995+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:38.188 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:38.182+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:38.189 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:38.186+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:38.343 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:38.337+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:38.413 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:38.405+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:38.674 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:38.670+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:38.675 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:38.673+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:38.722 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:38.716+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:38.842 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:38.837+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:38.981 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:38.974+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:39.225 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:39.205+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:39.225 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:39.206+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:39.326 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:39.320+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:39.365 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:39.360+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:39.701 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:39.695+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:39.702 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:39.699+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:39.708 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:39.702+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:39.808 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:39.803+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:39.942 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:39.936+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:40.218 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:40.214+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:40.220 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:40.218+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:40.344 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:40.337+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:40.365 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:40.361+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:40.704 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:40.698+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:40.752 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:40.748+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:40.755 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:40.752+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:40.767 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:40.763+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:40.981 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:40.977+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:41.239 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:41.237+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:41.240 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:41.239+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:41.326 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:41.321+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:41.378 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:41.373+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:41.756 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:41.745+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:41.759 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:41.750+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:41.780 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:41.776+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:41.786 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:41.783+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:41.949 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:41.943+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:42.248 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:42.244+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:42.249 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:42.246+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:42.296 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:42.292+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:42.425 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:42.420+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:42.722 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:42.709+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:42.734 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:42.727+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:42.749 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:42.745+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:42.751 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:42.748+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:42.963 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:42.957+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:43.213 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:43.208+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:43.215 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:43.211+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:43.292 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:43.288+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:43.443 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:43.436+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:43.565 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:13:43.561+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e255 v255) 2019-12-10T02:13:43.701 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:43.697+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:43.703 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:43.700+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:43.704 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:43.702+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:43.781 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:43.772+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:43.947 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:43.942+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:44.246 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:44.243+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:44.264 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:44.260+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:44.265 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:44.262+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:44.451 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:44.445+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:44.718 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:44.714+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:44.722 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:44.720+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:44.737 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:44.731+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:44.744 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:44.738+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:44.976 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:44.970+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:45.294 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:45.291+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:45.309 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:45.305+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:45.310 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:45.307+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:45.487 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:45.481+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:45.679 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:45.673+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:45.680 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:45.676+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:45.701 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:45.688+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:45.768 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:45.762+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:46.008 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:46.000+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:46.301 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:46.297+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:46.301 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:46.299+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:46.320 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:46.317+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:46.484 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:46.475+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:46.668 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:46.663+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:46.717 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:46.714+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:46.719 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:46.717+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:46.810 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:46.805+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:47.022 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:47.017+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:47.286 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:47.282+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:47.306 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:47.302+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:47.308 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:47.304+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:47.454 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:47.447+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:47.717 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:47.711+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:47.736 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:47.724+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:47.738 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:47.726+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:47.812 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:47.806+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:48.009 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:48.003+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:48.335 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:48.331+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:48.356 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:48.352+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:48.357 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:48.355+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:48.483 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:48.477+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:48.581 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:13:48.577+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e255 v255) 2019-12-10T02:13:48.724 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:48.721+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:48.728 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:48.726+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:48.768 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:48.762+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:48.853 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:48.847+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:49.045 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:49.039+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:49.362 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:49.358+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:49.391 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:49.386+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:49.391 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:49.389+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:49.454 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:49.449+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:49.699 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:49.694+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:49.700 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:49.697+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:49.813 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:49.805+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:49.822 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:49.817+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:50.014 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:50.008+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:50.411 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:50.404+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:50.416 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:50.413+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:50.417 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:50.413+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:50.420 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:50.417+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:50.742 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:50.739+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:50.745 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:50.743+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:50.840 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:50.834+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:50.872 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:50.867+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:51.035 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:51.029+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:51.392 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:51.388+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:51.393 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:51.390+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:51.403 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:51.397+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:51.407 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:51.402+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:51.731 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:51.725+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:51.732 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:51.729+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:51.810 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:51.804+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:51.872 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:51.866+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:52.077 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:52.072+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:52.360 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:52.357+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:52.362 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:52.360+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:52.377 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:52.371+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:52.399 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:52.397+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:52.750 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:52.747+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:52.752 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:52.751+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:52.774 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:52.768+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:52.860 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:52.853+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:53.041 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:53.036+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:53.384 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:53.377+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:53.396 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:53.392+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:53.402 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:53.398+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:53.402 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:53.400+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:53.602 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:13:53.600+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e255 v255) 2019-12-10T02:13:53.719 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:53.713+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:53.720 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:53.715+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:53.812 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:53.806+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:53.823 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:53.818+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:54.004 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:53.998+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:54.404 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:54.398+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:54.428 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:54.423+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:54.449 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:54.445+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:54.451 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:54.449+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:54.685 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:54.680+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:54.686 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:54.683+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:54.770 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:54.764+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:54.850 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:54.844+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:55.022 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:55.016+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:55.408 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:55.404+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:55.409 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:55.406+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:55.449 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:55.443+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:55.452 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:55.448+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:55.651 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:55.647+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:55.655 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:55.652+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:55.791 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:55.784+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:55.874 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:55.868+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:56.074 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:56.068+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:56.432 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:56.426+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:56.449 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:56.446+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:56.450 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:56.447+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:56.501 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:56.497+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:56.692 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:56.687+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:56.696 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:56.690+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:56.805 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:56.794+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:56.892 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:56.888+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:57.032 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:57.027+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:57.483 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:57.479+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:57.485 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:57.482+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:57.496 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:57.492+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:57.500 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:57.484+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:57.673 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:57.669+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:57.675 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:57.672+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:57.791 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:57.785+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:57.921 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:57.914+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:57.999 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:57.993+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:58.458 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:58.452+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:58.522 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:58.516+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:58.527 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:58.523+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:58.527 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:58.526+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:58.630 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:13:58.624+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e255 v255) 2019-12-10T02:13:58.690 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:58.685+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:58.690 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:58.687+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:58.830 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:58.825+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:58.963 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:58.956+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:58.972 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:58.965+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:13:59.428 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:13:59.424+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:13:59.483 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:13:59.477+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:13:59.492 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:59.488+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:13:59.494 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:13:59.490+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:13:59.718 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:59.713+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:13:59.719 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:13:59.716+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:13:59.862 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:13:59.854+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:13:59.991 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:13:59.985+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:13:59.999 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:13:59.993+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:14:00.451 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:14:00.447+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:14:00.453 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:14:00.450+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:14:00.454 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:14:00.452+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:14:00.488 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:14:00.482+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:14:00.764 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:14:00.761+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:14:00.767 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:14:00.765+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:14:00.841 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:14:00.834+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:14:00.948 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:14:00.941+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:14:00.991 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:14:00.986+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:14:01.413 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:14:01.409+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:14:01.474 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:14:01.466+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:14:01.488 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:14:01.485+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:14:01.488 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:14:01.487+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:14:01.759 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:14:01.757+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:14:01.760 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:14:01.758+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:14:01.864 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:14:01.858+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:14:01.920 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:14:01.913+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:14:01.972 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:14:01.966+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:14:02.370 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:14:02.367+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:14:02.502 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:14:02.499+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:14:02.502 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:14:02.501+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:14:02.522 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:14:02.516+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:14:02.740 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:14:02.739+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:14:02.742 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:14:02.741+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:14:02.830 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:14:02.823+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:14:02.904 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:14:02.898+0000 1694f700 -1 osd.7 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.502656+0000 front 2019-12-10T02:12:29.467348+0000 (oldest deadline 2019-12-10T02:13:12.325718+0000) 2019-12-10T02:14:02.958 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:14:02.952+0000 1694f700 -1 osd.4 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.765858+0000 front 2019-12-10T02:12:28.764146+0000 (oldest deadline 2019-12-10T02:13:12.241516+0000) 2019-12-10T02:14:03.400 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:14:03.395+0000 1694f700 -1 osd.0 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.504114+0000 front 2019-12-10T02:12:29.543917+0000 (oldest deadline 2019-12-10T02:13:14.742930+0000) 2019-12-10T02:14:03.491 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:14:03.485+0000 1694f700 -1 osd.6 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:30.743639+0000 front 2019-12-10T02:12:30.891858+0000 (oldest deadline 2019-12-10T02:13:11.703154+0000) 2019-12-10T02:14:03.517 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:14:03.514+0000 1694f700 -1 osd.2 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:29.599551+0000 front 2019-12-10T02:12:29.253894+0000 (oldest deadline 2019-12-10T02:13:14.509803+0000) 2019-12-10T02:14:03.519 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:14:03.517+0000 1694f700 -1 osd.2 255 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e247) 2019-12-10T02:14:03.712 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:14:03.708+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e255 v255) 2019-12-10T02:14:03.761 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:14:03.756+0000 1694f700 -1 osd.3 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:12:48.480969+0000 (oldest deadline 2019-12-10T02:13:28.480969+0000) 2019-12-10T02:14:03.762 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:14:03.760+0000 1694f700 -1 osd.3 255 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e253) 2019-12-10T02:14:03.864 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:14:03.859+0000 1694f700 -1 osd.5 255 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:12:28.492453+0000 front 2019-12-10T02:12:28.503962+0000 (oldest deadline 2019-12-10T02:13:10.954505+0000) 2019-12-10T02:14:04.730 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:14:04.596+0000 31f87700 -1 osd.3 256 build_incremental_map_msg missing incremental map 254 2019-12-10T02:14:35.766 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:14:35.704+0000 35f8f700 -1 osd.1 260 build_incremental_map_msg missing incremental map 260 2019-12-10T02:14:35.789 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:14:35.714+0000 35f8f700 -1 osd.1 260 build_incremental_map_msg missing incremental map 260 2019-12-10T02:14:36.030 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:14:35.942+0000 35f8f700 -1 osd.1 260 build_incremental_map_msg missing incremental map 260 2019-12-10T02:15:05.060 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:05.056+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:06.099 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:06.097+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:07.129 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:07.126+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:08.121 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:08.115+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:09.082 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:09.080+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:10.088 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:10.086+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:11.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:11.052+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:12.094 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:12.092+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:13.130 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:13.127+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:14.112 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:14.109+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:15.118 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:15.115+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:16.157 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:16.154+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:17.123 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:17.118+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:18.093 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:18.091+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:19.076 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:19.073+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:20.041 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:20.040+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:21.064 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:21.061+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:22.056 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:22.053+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:23.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:23.009+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:23.969 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:23.964+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:24.993 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:24.991+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:25.978 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:25.975+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:26.942 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:26.940+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:27.989 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:27.986+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:29.017 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:29.014+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:30.033 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:30.031+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:31.000 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:30.999+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:31.955 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:31.954+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:32.925 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:32.923+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:33.958 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:33.955+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:34.947 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:34.941+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:34.952 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:34.950+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:35.773 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:35.766+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:35.921 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:35.919+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:35.922 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:35.921+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:35.925 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:35.919+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:36.471 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:36.461+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:36.742 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:36.739+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:36.876 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:36.869+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:36.939 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:36.936+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:36.939 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:36.938+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:36.965 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:36.951+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:37.430 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:37.425+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:37.771 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:37.769+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:37.896 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:37.890+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:37.924 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:37.922+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:37.925 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:37.923+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:37.949 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:37.944+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:38.047 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:38.045+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:38.444 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:38.439+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:38.797 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:38.792+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:38.902 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:38.895+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:38.934 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:38.932+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:38.934 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:38.933+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:38.937 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:38.931+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:39.066 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:39.065+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:39.479 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:39.473+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:39.845 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:39.838+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:39.952 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:39.939+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:39.952 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:39.940+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:39.993 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:39.985+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:39.993 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:39.988+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:40.079 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:40.075+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:40.453 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:40.445+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:40.885 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:40.882+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:40.912 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:40.905+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:40.914 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:40.909+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:40.989 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:40.986+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:40.990 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:40.988+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:41.072 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:41.068+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:41.411 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:41.404+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:41.913 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:41.907+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:41.916 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:41.915+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:41.917 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:41.911+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:41.950 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:41.945+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:41.951 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:41.949+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:42.113 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:42.111+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:42.366 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:42.360+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:42.886 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:42.880+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:42.899 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:42.896+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:42.954 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:42.949+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:42.986 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:42.983+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:42.987 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:42.985+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:43.118 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:43.114+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:43.369 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:43.351+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:43.843 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:43.837+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:43.947 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:43.944+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:43.981 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:43.973+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:44.021 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:44.015+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:44.021 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:44.016+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:44.125 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:44.122+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:44.344 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:44.335+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:44.851 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:44.845+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:44.923 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:44.919+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:45.017 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:45.014+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:45.018 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:45.015+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:45.027 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:45.022+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:45.103 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:45.101+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:45.374 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:45.368+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:45.851 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:45.846+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:45.903 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:45.899+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:45.988 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:45.984+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:45.988 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:45.986+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:46.025 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:46.019+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:46.100 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:46.098+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:46.386 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:46.380+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:46.810 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:46.805+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:46.882 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:46.878+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:47.037 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:47.032+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:47.037 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:47.035+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:47.056 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:47.050+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:47.087 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:47.083+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:47.346 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:47.341+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:47.837 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:47.829+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:47.873 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:47.866+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:48.050 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:48.046+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:48.051 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:48.048+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:48.076 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:48.071+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:48.106 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:48.099+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:48.329 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:48.318+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:48.846 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:48.842+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:48.853 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:48.846+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:49.019 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:49.016+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:49.020 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:49.018+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:49.086 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:49.080+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:49.116 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:49.109+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:49.377 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:49.370+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:49.843 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:49.840+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:49.849 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:49.844+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:50.004 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:49.996+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:50.004 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:49.998+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:50.123 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:50.120+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:50.131 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:50.125+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:50.426 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:50.419+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:50.847 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:50.843+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:50.885 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:50.876+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:51.004 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:50.999+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:51.005 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:51.001+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:51.095 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:51.085+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:51.154 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:51.151+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:51.477 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:51.468+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:51.899 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:51.894+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:51.921 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:51.908+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:52.000 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:51.997+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:52.000 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:51.998+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:52.108 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:52.102+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:52.108 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:52.106+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:52.493 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:52.486+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:52.854 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:52.850+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:52.881 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:52.874+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:53.021 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:53.017+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:53.022 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:53.019+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:53.116 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:53.111+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:53.119 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:53.112+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:53.451 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:53.444+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:53.829 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:53.818+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:53.916 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:53.910+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:54.044 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:54.033+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:54.044 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:54.035+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:54.101 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:54.084+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:54.163 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:54.161+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:54.411 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:54.405+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:54.869 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:54.866+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:54.941 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:54.935+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:55.039 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:55.036+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:55.039 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:55.037+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:55.062 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:55.057+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:55.122 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:55.120+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:55.461 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:55.455+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:55.905 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:55.902+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:55.915 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:55.910+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:56.020 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:56.016+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:56.021 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:56.018+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:56.044 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:56.039+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:56.089 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:56.086+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:56.423 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:56.418+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:56.930 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:56.926+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:56.959 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:56.953+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:57.055 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:57.052+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:57.056 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:57.054+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:57.097 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:57.091+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:57.101 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:57.098+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:57.417 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:57.412+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:57.884 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:57.881+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:57.973 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:57.967+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:58.028 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:58.024+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:58.029 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:58.026+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:58.110 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:58.104+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:58.142 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:58.139+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:58.443 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:58.437+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:58.941 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:58.930+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:15:58.984 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:15:58.979+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:15:59.065 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:59.063+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:15:59.066 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:15:59.064+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:15:59.125 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:15:59.119+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:15:59.129 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:15:59.127+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:15:59.407 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:15:59.400+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:15:59.892 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:15:59.889+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:00.022 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:00.018+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:00.022 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:00.020+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:00.028 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:00.022+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:00.089 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:00.083+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:00.165 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:00.161+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:00.376 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:00.370+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:00.873 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:00.869+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:01.005 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:00.998+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:01.016 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:01.013+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:01.016 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:01.014+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:01.096 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:01.091+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:01.202 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:01.199+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:01.338 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:01.333+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:01.858 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:01.855+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:01.976 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:01.973+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:01.976 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:01.974+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:01.981 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:01.976+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:02.148 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:02.142+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:02.160 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:02.157+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:02.329 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:02.323+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:02.902 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:02.897+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:02.984 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:02.980+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:02.985 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:02.982+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:03.038 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:03.032+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:03.109 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:03.103+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:03.213 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:03.208+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:03.363 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:03.357+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:03.894 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:03.891+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:03.975 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:03.970+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:03.976 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:03.973+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:04.069 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:04.064+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:04.100 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:04.096+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:04.255 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:04.252+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:04.356 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:04.351+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:04.736 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:04.729+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:04.897 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:04.894+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:05.017 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:05.014+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:05.018 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:05.015+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:05.063 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:05.058+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:05.085 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:05.080+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:05.241 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:05.237+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:05.350 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:05.343+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:05.760 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:05.755+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:05.916 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:05.913+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:06.039 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:06.036+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:06.040 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:06.037+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:06.041 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:06.036+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:06.090 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:06.084+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:06.213 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:06.209+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:06.344 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:06.338+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:06.759 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:06.753+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:06.945 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:06.941+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:07.014 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:07.010+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:07.015 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:07.012+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:07.041 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:07.035+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:07.103 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:07.098+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:07.236 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:07.233+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:07.329 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:07.324+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:07.777 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:07.772+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:07.963 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:07.960+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:08.006 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:08.000+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:08.008 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:08.006+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:08.009 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:08.007+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:08.144 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:08.139+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:08.259 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:08.255+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:08.348 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:08.342+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:08.810 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:08.804+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:08.962 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:08.956+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:08.979 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:08.975+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:09.063 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:09.058+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:09.063 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:09.060+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:09.150 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:09.144+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:09.168 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:16:09.166+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 42sec e261 v261) 2019-12-10T02:16:09.288 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:09.285+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:09.310 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:09.306+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:09.804 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:09.799+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:09.957 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:09.954+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:09.975 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:09.970+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:10.090 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:10.086+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:10.091 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:10.088+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:10.122 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:10.116+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:10.276 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:10.270+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:10.337 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:10.334+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:10.839 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:10.834+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:10.948 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:10.942+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:10.988 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:10.985+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:11.079 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:11.075+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:11.079 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:11.077+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:11.103 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:11.096+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:11.283 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:11.277+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:11.302 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:11.297+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:11.840 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:11.835+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:11.906 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:11.900+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:11.961 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:11.957+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:12.079 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:12.076+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:12.079 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:12.077+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:12.140 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:12.134+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:12.292 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:12.289+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:12.315 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:12.310+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:12.867 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:12.861+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:12.881 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:12.875+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:12.961 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:12.958+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:13.078 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:13.075+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:13.079 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:13.076+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:13.182 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:13.177+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:13.285 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:13.276+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:13.357 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:13.352+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:13.845 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:16:13.839+0000 1694f700 -1 osd.6 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:54.507476+0000 front 2019-12-10T02:14:53.777143+0000 (oldest deadline 2019-12-10T02:15:36.148629+0000) 2019-12-10T02:16:13.878 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:13.873+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:13.945 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:16:13.942+0000 1694f700 -1 osd.1 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.521282+0000 front 2019-12-10T02:14:52.229483+0000 (oldest deadline 2019-12-10T02:15:35.081883+0000) 2019-12-10T02:16:14.061 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:14.058+0000 1694f700 -1 osd.2 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:51.217985+0000 front 2019-12-10T02:14:51.228530+0000 (oldest deadline 2019-12-10T02:15:35.811594+0000) 2019-12-10T02:16:14.062 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:14.060+0000 1694f700 -1 osd.2 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:14.215 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:16:14.209+0000 1694f700 -1 osd.7 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769506+0000 front 2019-12-10T02:14:54.436807+0000 (oldest deadline 2019-12-10T02:15:34.936706+0000) 2019-12-10T02:16:14.230 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:16:14.227+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 42sec e261 v261) 2019-12-10T02:16:14.268 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:16:14.262+0000 1694f700 -1 osd.0 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.542001+0000 front 2019-12-10T02:14:53.419713+0000 (oldest deadline 2019-12-10T02:15:37.143817+0000) 2019-12-10T02:16:14.346 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:16:14.330+0000 1694f700 -1 osd.4 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:53.769162+0000 front 2019-12-10T02:14:53.823430+0000 (oldest deadline 2019-12-10T02:15:36.122364+0000) 2019-12-10T02:16:14.869 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:14.865+0000 1694f700 -1 osd.5 261 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:14:52.396104+0000 front 2019-12-10T02:14:52.408684+0000 (oldest deadline 2019-12-10T02:16:03.873350+0000) 2019-12-10T02:16:15.051 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:15.047+0000 1694f700 -1 osd.2 262 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:16.089 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:16.076+0000 1694f700 -1 osd.2 263 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:17.041 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:17.040+0000 1694f700 -1 osd.2 263 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:107037 2.3as0 2:5dc98bb7:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_6341062ab8784ce5af537a870960bc06:head [cmpxattr user.rgw.idtag (48) op 1 mode 1 in=62b,call rgw.obj_remove in=40b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e258) 2019-12-10T02:16:51.085 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:51.071+0000 31786700 -1 osd.2 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.087 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:51.076+0000 31786700 -1 osd.2 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.151 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:51.135+0000 37f93700 -1 osd.2 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.183 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:51.034+0000 32f89700 -1 osd.5 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.195 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:51.054+0000 32f89700 -1 osd.5 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.297 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:51.132+0000 36790700 -1 osd.5 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.298 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:51.133+0000 36790700 -1 osd.5 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.453 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:51.249+0000 35f8f700 -1 osd.5 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.543 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:51.359+0000 35f8f700 -1 osd.2 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.639 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:16:51.430+0000 35f8f700 -1 osd.5 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.693 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:51.484+0000 36790700 -1 osd.2 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:16:51.745 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:16:51.675+0000 3478c700 -1 osd.2 265 build_incremental_map_msg missing incremental map 265 2019-12-10T02:17:04.205 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:43:25.268 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:04.211 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:43:25.275 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:04.228 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:43:25.291 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:04.233 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:43:25.297 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:04.234 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:43:25.394 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:04.239 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:43:25.398 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:04.483 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:43:25.545 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:04.494 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:43:25.654 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:04.587 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:43:25.650 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:12.524 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:12.520+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:13.391 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:13.386+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:13.509 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:13.505+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:13.783 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:13.777+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:14.123 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:14.120+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:14.362 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:14.356+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:14.504 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:14.501+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:14.772 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:14.766+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:15.171 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:15.167+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:15.360 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:15.353+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:15.474 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:15.468+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:15.478 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:15.476+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:15.747 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:15.741+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:16.128 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:16.124+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:16.320 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:16.314+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:16.442 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:16.439+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:16.493 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:16.487+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:16.743 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:16.735+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:17.098 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:17.094+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:17.339 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:17.334+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:17.462 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:17.459+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:17.484 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:17.478+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:17.616 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:17.610+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:17.747 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:17.739+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:18.055 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:18.052+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:18.381 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:18.376+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:18.461 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:18.443+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:18.510 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:18.506+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:18.635 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:18.629+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:18.742 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:18.732+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:19.084 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:19.079+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:19.348 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:19.343+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:19.463 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:19.457+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:19.528 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:19.525+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:19.615 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:19.609+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:19.698 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:19.692+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:20.034 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:43:41.097 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:20.059 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:20.056+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:20.071 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:43:41.135 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:20.086 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:43:41.149 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:20.107 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:43:41.170 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:20.115 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:43:41.178 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:20.150 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:43:41.309 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:20.154 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:43:41.313 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:20.159 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:43:41.223 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:20.201 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:43:41.360 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:17:20.389 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:20.381+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:20.495 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:20.489+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:20.568 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:20.565+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:20.577 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:20.572+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:20.738 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:20.725+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:21.022 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:21.008+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:21.360 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:21.353+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:21.504 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:21.499+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:21.563 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:21.560+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:21.578 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:21.572+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:21.771 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:21.765+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:22.054 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:22.049+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:22.394 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:22.386+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:22.559 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:22.554+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:22.560 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:22.551+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:22.562 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:22.556+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:22.776 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:22.770+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:22.806 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:22.803+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:23.049 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:23.046+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:23.351 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:23.345+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:23.544 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:23.539+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:23.598 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:23.593+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:23.606 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:23.600+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:23.732 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:23.725+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:23.789 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:23.786+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:24.009 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:24.007+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:24.325 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:24.319+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:24.551 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:24.546+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:24.567 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:24.565+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:24.616 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:24.610+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:24.694 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:24.683+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:24.843 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:24.840+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:25.025 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:25.021+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:25.319 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:25.314+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:25.519 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:25.509+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:25.593 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:25.589+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:25.605 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:25.600+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:25.704 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:25.698+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:25.882 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:25.879+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:25.987 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:25.984+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:26.322 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:26.316+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:26.492 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:26.487+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:26.569 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:26.564+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:26.641 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:26.638+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:26.669 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:26.659+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:26.864 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:26.862+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:27.020 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:27.016+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:27.285 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:27.280+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:27.497 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:27.492+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:27.567 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:27.562+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:27.648 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:27.645+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:27.697 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:27.691+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:27.908 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:27.902+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:28.051 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:28.048+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:28.301 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:28.294+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:28.477 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:28.472+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:28.537 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:28.530+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:28.663 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:28.657+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:28.664 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:28.662+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:28.888 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:28.880+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:28.889 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:28.886+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:29.010 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:29.007+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:29.257 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:29.250+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:29.481 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:29.475+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:29.553 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:29.546+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:29.664 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:29.660+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:29.712 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:29.686+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:29.847 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:29.840+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:29.849 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:29.845+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:30.031 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:30.026+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:30.252 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:30.247+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:30.477 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:30.470+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:30.577 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:30.570+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:30.683 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:30.679+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:30.699 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:30.691+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:30.818 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:30.813+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:30.825 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:30.819+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:31.052 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:31.049+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:31.296 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:31.291+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:31.481 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:31.471+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:31.620 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:31.613+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:31.716 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:31.711+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:31.735 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:31.729+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:31.829 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:31.815+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:31.829 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:31.820+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:32.087 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:32.084+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:32.338 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:32.332+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:32.435 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:32.429+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:32.670 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:32.666+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:32.701 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:32.696+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:32.759 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:32.756+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:32.859 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:32.856+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:32.862 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:32.860+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:33.110 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:33.105+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:33.297 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:33.291+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:33.420 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:33.415+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:33.656 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:33.649+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:33.661 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:33.655+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:33.786 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:33.781+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:33.834 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:33.829+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:33.835 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:33.832+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:34.150 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:34.147+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:34.317 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:34.311+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:34.422 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:34.416+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:34.666 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:34.656+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:34.703 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:34.665+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:34.827 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:34.825+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:34.878 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:34.875+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:34.880 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:34.876+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:35.126 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:35.122+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:35.310 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:35.304+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:35.432 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:35.427+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:35.686 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:35.680+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:35.715 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:35.709+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:35.791 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:35.785+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:35.875 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:35.871+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:35.877 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:35.873+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:36.151 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:36.147+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:36.339 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:36.333+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:36.407 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:36.400+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:36.688 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:36.682+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:36.704 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:36.697+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:36.754 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:36.750+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:36.930 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:36.925+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:36.932 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:36.928+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:37.125 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:37.120+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:37.351 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:37.344+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:37.432 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:37.425+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:37.694 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:37.688+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:37.738 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:37.732+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:37.741 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:37.736+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:37.937 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:37.934+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:37.938 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:37.936+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:38.149 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:38.145+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:38.344 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:38.338+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:38.453 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:38.446+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:38.675 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:38.669+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:38.733 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:38.727+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:38.741 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:38.737+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:38.895 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:38.891+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:38.896 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:38.893+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:39.138 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:39.135+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:39.359 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:39.353+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:39.447 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:39.441+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:39.658 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:39.652+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:39.710 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:39.705+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:39.740 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:39.733+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:39.947 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:39.942+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:39.949 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:39.945+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:40.162 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:40.159+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:40.331 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:40.327+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:40.459 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:40.452+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:40.646 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:40.640+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:40.696 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:40.693+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:40.723 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:40.712+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:40.968 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:40.964+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:40.972 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:40.966+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:41.211 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:41.207+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:41.296 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:41.289+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:41.481 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:41.476+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:41.658 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:41.653+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:41.683 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:41.680+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:41.771 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:41.764+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:41.970 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:41.963+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:41.971 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:41.966+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:42.177 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:42.174+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:42.329 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:42.324+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:42.532 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:42.520+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:42.702 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:42.699+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:42.705 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:42.699+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:42.773 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:42.766+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:42.961 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:42.958+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:42.961 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:42.959+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:43.133 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:43.128+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:43.296 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:43.290+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:43.572 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:43.566+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:43.683 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:43.679+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:43.738 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:43.732+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:43.826 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:43.819+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:43.959 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:43.953+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:43.963 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:43.957+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:44.144 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:44.140+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:44.274 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:44.268+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:44.604 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:44.598+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:44.641 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:44.638+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:44.780 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:44.774+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:44.861 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:44.855+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:44.977 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:44.974+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:44.981 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:44.978+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:45.077 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:17:45.065+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e267 v267) 2019-12-10T02:17:45.104 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:45.102+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:45.253 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:45.247+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:45.558 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:45.553+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:45.677 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:45.671+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:45.775 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:45.766+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:45.864 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:45.857+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:45.935 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:45.932+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:45.938 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:45.937+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:46.141 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:46.138+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:46.246 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:46.240+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:46.584 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:46.578+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:46.712 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:46.709+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:46.731 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:46.725+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:46.826 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:46.819+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:46.911 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:46.908+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:46.912 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:46.910+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:47.116 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:47.113+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:47.297 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:47.292+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:47.628 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:47.617+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:47.750 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:47.744+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:47.750 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:47.747+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:47.781 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:47.774+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:47.957 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:47.953+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:47.958 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:47.956+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:48.157 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:48.155+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:48.320 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:48.315+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:48.618 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:48.613+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:48.717 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:48.716+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:48.772 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:48.766+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:48.782 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:48.775+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:48.936 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:48.931+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:48.937 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:48.934+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:49.139 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:49.136+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:49.359 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:49.353+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:49.606 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:49.599+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:49.711 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:49.708+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:49.815 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:49.809+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:49.829 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:49.822+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:49.919 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:49.914+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:49.925 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:49.920+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:50.089 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:17:50.087+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e267 v267) 2019-12-10T02:17:50.112 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:50.109+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:50.357 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:50.351+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:50.622 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:50.616+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:50.683 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:50.682+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:50.842 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:50.836+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:50.868 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:50.861+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:50.940 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:50.936+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:50.941 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:50.939+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:51.132 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:51.128+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:51.322 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:51.317+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:51.616 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:51.610+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:51.729 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:51.726+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:51.807 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:51.801+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:51.838 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:51.831+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:51.981 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:51.978+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:51.981 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:51.980+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:52.137 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:52.134+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:52.377 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:52.371+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:52.657 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:52.650+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:52.735 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:52.732+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:52.790 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:52.784+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:52.804 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:52.798+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:53.007 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:53.004+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:53.012 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:53.010+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:53.191 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:53.188+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:53.405 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:53.398+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:53.406 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:53.401+0000 1694f700 -1 osd.5 267 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108155 2.2s0 2.3e1de382 (undecoded) ondisk+read+known_if_redirected e267) 2019-12-10T02:17:53.663 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:53.658+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:53.731 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:53.727+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:53.755 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:53.749+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:53.844 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:53.837+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:54.042 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:54.039+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:54.043 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:54.041+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:54.170 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:54.162+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:54.383 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:54.376+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:54.384 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:54.379+0000 1694f700 -1 osd.5 267 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108155 2.2s0 2.3e1de382 (undecoded) ondisk+read+known_if_redirected e267) 2019-12-10T02:17:54.651 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:54.642+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:54.733 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:54.727+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:54.786 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:54.778+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:54.895 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:54.889+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:55.035 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:55.030+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:55.036 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:55.033+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:55.136 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:17:55.131+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e267 v267) 2019-12-10T02:17:55.136 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:55.134+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:55.375 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:55.369+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:55.376 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:55.371+0000 1694f700 -1 osd.5 267 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108155 2.2s0 2.3e1de382 (undecoded) ondisk+read+known_if_redirected e267) 2019-12-10T02:17:55.617 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:55.611+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:55.692 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:55.686+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:55.811 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:55.809+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:55.912 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:55.906+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:55.999 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:55.998+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:56.001 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:56.000+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:56.127 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:56.125+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:56.366 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:56.362+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:56.366 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:56.363+0000 1694f700 -1 osd.5 267 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108155 2.2s0 2.3e1de382 (undecoded) ondisk+read+known_if_redirected e267) 2019-12-10T02:17:56.668 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:56.661+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:56.745 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:56.739+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:56.808 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:56.799+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:56.906 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:56.900+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:56.980 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:56.977+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:56.981 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:56.979+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:57.108 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:57.105+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:57.411 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:57.404+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:57.416 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:57.406+0000 1694f700 -1 osd.5 267 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108155 2.2s0 2.3e1de382 (undecoded) ondisk+read+known_if_redirected e267) 2019-12-10T02:17:57.697 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:57.691+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:57.788 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:57.781+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:57.793 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:57.789+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:57.937 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:57.931+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:58.001 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:57.998+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:58.002 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:58.001+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:58.096 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:58.093+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:58.389 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:58.382+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:58.390 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:58.385+0000 1694f700 -1 osd.5 267 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108155 2.2s0 2.3e1de382 (undecoded) ondisk+read+known_if_redirected e267) 2019-12-10T02:17:58.701 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:58.695+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:58.751 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:58.748+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:58.776 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:58.769+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:58.944 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:58.935+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:58.987 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:58.985+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:58.991 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:58.989+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:17:59.070 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:17:59.067+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:17:59.431 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:59.424+0000 1694f700 -1 osd.5 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:26.670252+0000 front 2019-12-10T02:16:26.666346+0000 (oldest deadline 2019-12-10T02:17:12.546266+0000) 2019-12-10T02:17:59.432 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:17:59.427+0000 1694f700 -1 osd.5 267 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108155 2.2s0 2.3e1de382 (undecoded) ondisk+read+known_if_redirected e267) 2019-12-10T02:17:59.661 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:17:59.653+0000 1694f700 -1 osd.4 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.563399+0000 front 2019-12-10T02:16:31.730605+0000 (oldest deadline 2019-12-10T02:17:17.400289+0000) 2019-12-10T02:17:59.761 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:17:59.755+0000 1694f700 -1 osd.7 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.188738+0000 front 2019-12-10T02:16:28.195758+0000 (oldest deadline 2019-12-10T02:17:13.086126+0000) 2019-12-10T02:17:59.806 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:17:59.800+0000 1694f700 -1 osd.2 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:27.629525+0000 front 2019-12-10T02:16:27.552368+0000 (oldest deadline 2019-12-10T02:17:12.234315+0000) 2019-12-10T02:17:59.901 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:17:59.895+0000 1694f700 -1 osd.6 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:31.581796+0000 front 2019-12-10T02:16:31.584369+0000 (oldest deadline 2019-12-10T02:17:15.448238+0000) 2019-12-10T02:17:59.957 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:59.953+0000 1694f700 -1 osd.3 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:16:48.174456+0000 (oldest deadline 2019-12-10T02:17:28.174456+0000) 2019-12-10T02:17:59.959 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:17:59.955+0000 1694f700 -1 osd.3 267 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4557.0:106134 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e265) 2019-12-10T02:18:00.108 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:18:00.105+0000 1694f700 -1 osd.0 267 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:16:28.880578+0000 front 2019-12-10T02:16:29.367506+0000 (oldest deadline 2019-12-10T02:17:13.524176+0000) 2019-12-10T02:18:00.188 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:18:00.182+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e267 v267) 2019-12-10T02:19:34.632 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:34.627+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:35.237 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:35.228+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:35.662 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:35.655+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:35.663 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:35.658+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:36.213 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:36.208+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:36.246 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:36.241+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:36.589 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:36.586+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:36.628 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:36.622+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:36.628 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:36.624+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:37.228 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:37.223+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:37.277 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:37.274+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:37.555 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:37.550+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:37.635 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:37.627+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:37.636 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:37.629+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:37.641 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:37.638+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:38.264 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:38.259+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:38.332 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:38.324+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:38.532 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:38.527+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:38.648 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:38.644+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:38.664 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:38.659+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:38.667 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:38.663+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:39.313 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:39.303+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:39.366 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:39.362+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:39.378 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:39.373+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:39.554 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:39.548+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:39.645 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:39.639+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:39.648 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:39.644+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:39.675 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:39.671+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:40.274 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:40.268+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:40.375 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:40.368+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:40.410 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:40.407+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:40.528 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:40.523+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:40.629 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:40.625+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:40.636 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:40.630+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:40.637 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:40.632+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:41.298 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:41.291+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:41.389 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:41.384+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:41.422 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:41.417+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:41.512 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:41.506+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:41.657 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:41.652+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:41.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:41.678+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:41.685 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:41.680+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:42.278 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:42.272+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:42.377 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:42.372+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:42.381 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:42.375+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:42.562 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:42.556+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:42.637 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:42.633+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:42.685 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:42.679+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:42.686 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:42.681+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:43.264 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:43.258+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:43.411 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:43.407+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:43.432 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:43.426+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:43.559 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:43.554+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:43.636 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:43.631+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:43.663 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:43.657+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:43.663 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:43.659+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:44.324 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:44.299+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:44.389 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:44.384+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:44.412 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:44.406+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:44.530 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:44.524+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:44.645 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:44.642+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:44.700 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:44.695+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:44.703 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:44.699+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:45.347 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:45.341+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:45.374 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:45.371+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:45.389 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:45.377+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:45.539 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:45.533+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:45.709 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:45.696+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:45.709 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:45.699+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:45.709 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:45.692+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:46.332 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:46.326+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:46.381 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:46.378+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:46.382 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:46.376+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:46.505 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:46.499+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:46.664 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:46.658+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:46.665 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:46.660+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:46.705 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:46.703+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:47.293 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:47.288+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:47.389 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:47.386+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:47.431 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:47.422+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:47.461 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:47.456+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:47.646 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:47.639+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:47.646 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:47.642+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:47.720 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:47.716+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:48.316 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:48.310+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:48.432 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:48.429+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:48.456 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:48.449+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:48.464 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:48.453+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:48.622 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:48.618+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:48.623 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:48.619+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:48.705 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:48.701+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:49.332 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:49.325+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:49.473 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:49.469+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:49.476 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:49.470+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:49.489 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:49.482+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:49.649 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:49.643+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:49.650 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:49.645+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:49.722 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:49.718+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:50.358 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:50.352+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:50.451 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:50.445+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:50.467 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:50.461+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:50.471 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:50.467+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:50.635 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:50.630+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:50.638 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:50.634+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:50.707 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:50.703+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:51.382 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:51.376+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:51.459 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:51.453+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:51.489 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:51.485+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:51.496 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:51.490+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:51.686 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:51.681+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:51.689 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:51.685+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:51.722 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:51.717+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:52.425 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:52.421+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:52.431 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:52.414+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:52.456 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:52.453+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:52.516 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:52.511+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:52.670 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:52.664+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:52.671 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:52.666+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:52.742 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:52.739+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:53.377 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:53.371+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:53.437 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:53.431+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:53.489 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:53.487+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:53.515 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:53.509+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:53.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:53.676+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:53.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:53.678+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:53.729 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:53.725+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:54.381 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:54.375+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:54.451 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:54.445+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:54.485 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:54.483+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:54.490 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:54.485+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:54.721 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:54.718+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:54.731 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:54.722+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:54.732 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:54.724+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:55.380 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:55.368+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:55.472 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:55.465+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:55.473 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:55.467+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:55.522 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:55.512+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:55.699 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:55.692+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:55.699 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:55.696+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:55.700 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:55.695+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:56.394 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:56.389+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:56.517 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:56.513+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:56.519 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:56.515+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:56.568 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:56.565+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:56.666 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:56.663+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:56.689 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:56.684+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:56.691 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:56.688+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:57.426 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:57.420+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:57.513 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:57.508+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:57.536 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:57.531+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:57.580 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:57.577+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:57.654 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:57.639+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:57.672 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:57.667+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:57.676 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:57.671+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:58.420 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:58.407+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:58.522 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:58.516+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:58.537 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:58.533+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:58.559 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:58.553+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:58.652 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:58.646+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:58.653 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:58.648+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:19:58.655 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:58.651+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:59.389 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:19:59.382+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:19:59.488 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:19:59.481+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:19:59.587 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:19:59.584+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:19:59.604 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:19:59.598+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:19:59.652 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:19:59.649+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:19:59.674 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:59.667+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:19:59.676 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:19:59.672+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:00.371 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:00.366+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:00.463 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:00.457+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:00.562 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:00.556+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:00.623 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:00.620+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:00.644 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:00.638+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:00.645 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:00.642+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:00.675 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:00.672+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:01.387 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:01.381+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:01.490 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:01.484+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:01.546 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:01.538+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:01.620 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:01.618+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:01.669 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:01.662+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:01.670 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:01.665+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:01.670 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:01.667+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:02.375 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:02.370+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:02.528 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:02.522+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:02.596 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:02.581+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:02.641 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:02.637+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:02.702 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:02.695+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:02.702 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:02.697+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:02.709 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:02.706+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:03.343 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:03.337+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:03.529 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:03.523+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:03.591 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:03.585+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:03.679 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:03.675+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:03.711 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:03.705+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:03.712 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:03.707+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:03.741 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:03.738+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:04.331 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:04.325+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:04.514 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:04.505+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:04.621 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:04.616+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:04.706 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:04.705+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:04.733 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:04.718+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:04.733 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:04.720+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:04.756 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:04.754+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:05.370 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:05.364+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:05.497 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:05.491+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:05.656 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:05.645+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:05.719 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:05.716+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:05.721 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:05.719+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:05.777 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:05.771+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:05.778 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:05.773+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:06.349 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:06.343+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:06.528 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:06.521+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:06.644 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:06.638+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:06.709 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:06.707+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:06.762 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:06.758+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:06.774 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:06.765+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:06.776 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:06.770+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:07.316 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:07.309+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:07.482 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:07.476+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:07.663 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:07.657+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:07.729 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:07.726+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:07.775 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:07.770+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:07.775 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:07.765+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:07.776 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:07.769+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:08.367 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:08.361+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:08.448 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:08.439+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:08.631 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:08.625+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:08.716 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:08.713+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:08.733 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:08.731+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:08.800 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:08.795+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:08.803 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:08.799+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:09.346 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:09.339+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:09.452 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:09.446+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:09.677 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:09.671+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:09.704 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:09.700+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:09.777 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:09.771+0000 1694f700 -1 osd.5 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.845031+0000 front 2019-12-10T02:18:53.838252+0000 (oldest deadline 2019-12-10T02:19:35.467409+0000) 2019-12-10T02:20:09.778 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:20:09.775+0000 1694f700 -1 osd.2 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:50.519166+0000 front 2019-12-10T02:18:50.487871+0000 (oldest deadline 2019-12-10T02:19:36.341889+0000) 2019-12-10T02:20:09.778 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:09.773+0000 1694f700 -1 osd.5 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:10.340 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:20:10.333+0000 1694f700 -1 osd.7 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:53.248283+0000 front 2019-12-10T02:18:53.482307+0000 (oldest deadline 2019-12-10T02:19:39.086089+0000) 2019-12-10T02:20:10.433 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:20:10.425+0000 1694f700 -1 osd.4 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:51.722963+0000 front 2019-12-10T02:18:51.736386+0000 (oldest deadline 2019-12-10T02:19:35.204643+0000) 2019-12-10T02:20:10.619 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:20:10.603+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 41sec e272 v272) 2019-12-10T02:20:10.647 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:20:10.637+0000 1694f700 -1 osd.6 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.535449+0000 front 2019-12-10T02:18:52.352270+0000 (oldest deadline 2019-12-10T02:19:36.999653+0000) 2019-12-10T02:20:10.742 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:20:10.739+0000 1694f700 -1 osd.0 272 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:18:52.560417+0000 front 2019-12-10T02:18:55.041414+0000 (oldest deadline 2019-12-10T02:19:34.783750+0000) 2019-12-10T02:20:11.258 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:11.143+0000 1694f700 -1 osd.5 273 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:12.644 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:20:12.639+0000 1694f700 -1 osd.5 274 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:108390 2.12s0 2:4af777af:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.665_cfdca7d47ef1479188c42490c8398fdd:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e272) 2019-12-10T02:20:55.858 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:47:16.921 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:20:55.863 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:47:16.926 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:20:55.872 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:47:16.935 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:20:55.880 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:47:16.940 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:20:55.884 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:47:17.043 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:20:55.894 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:47:17.053 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:20:56.150 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:47:17.309 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:20:56.179 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:47:17.242 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:20:56.203 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:47:17.266 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:21:09.252 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:09.247+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:10.067 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:10.061+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:10.264 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:10.257+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:11.058 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:11.052+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:11.218 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:11.214+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:11.954 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:11.948+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:12.060 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:12.053+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:12.177 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:12.171+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:12.582 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:12.578+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:12.961 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:12.956+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:13.060 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:13.054+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:13.143 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:13.135+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:13.558 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:13.554+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:13.906 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:13.903+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:13.946 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:13.941+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:14.028 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:14.020+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:14.102 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:14.093+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:14.593 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:14.587+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:14.870 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:14.864+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:14.975 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:14.968+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:15.021 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:15.015+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:15.081 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:15.075+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:15.571 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:15.567+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:15.910 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:15.907+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:15.998 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:15.993+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:16.045 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:16.039+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:16.068 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:16.062+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:16.529 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:16.526+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:16.903 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:16.900+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:17.006 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:17.000+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:17.011 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:17.005+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:17.113 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:17.108+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:17.508 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:17.505+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:17.954 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:17.950+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:17.983 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:17.977+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:18.043 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:18.035+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:18.102 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:18.096+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:18.529 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:18.525+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:18.940 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:18.929+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:19.021 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:19.014+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:19.040 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:19.032+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:19.155 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:19.149+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:19.490 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:19.486+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:19.962 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:19.957+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:20.003 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:19.996+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:20.069 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:20.060+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:20.142 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:20.136+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:20.479 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:20.476+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:20.972 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:20.965+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:20.985 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:20.974+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:21.086 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:21.080+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:21.168 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:21.162+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:21.511 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:21.507+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:21.766 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:21.763+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:21.954 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:21.950+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:22.001 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:21.995+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:22.060 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:22.053+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:22.137 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:22.129+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:22.470 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:22.466+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:22.791 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:22.787+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:22.974 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:22.969+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:22.978 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:22.971+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:23.091 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:23.085+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:23.175 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:23.168+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:23.479 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:23.462+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:23.808 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:23.806+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:23.995 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:23.991+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:24.019 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:24.013+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:24.056 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:24.050+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:24.139 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:24.133+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:24.429 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:24.425+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:24.841 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:24.836+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:25.017 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:25.011+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:25.027 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:25.021+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:25.045 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:25.040+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:25.151 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:25.145+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:25.471 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:25.467+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:25.854 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:25.850+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:25.979 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:25.972+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:26.037 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:26.032+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:26.056 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:26.050+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:26.148 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:26.142+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:26.514 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:26.511+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:26.819 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:26.815+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:26.945 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:26.939+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:27.018 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:27.012+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:27.039 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:27.037+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:27.200 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:27.192+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:27.475 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:27.472+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:27.798 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:27.793+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:27.799 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:27.796+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:27.965 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:27.959+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:28.026 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:28.017+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:28.067 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:28.060+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:28.193 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:28.188+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:28.514 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:28.511+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:28.817 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:28.813+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:28.818 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:28.816+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:29.003 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:28.998+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:29.049 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:29.043+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:29.097 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:29.096+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:29.151 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:29.145+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:29.473 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:29.468+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:29.824 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:29.820+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:29.825 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:29.822+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:30.027 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:30.020+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:30.031 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:30.024+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:30.084 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:30.082+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:30.148 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:30.143+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:30.474 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:30.461+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:30.790 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:30.784+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:30.791 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:30.787+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:31.056 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:31.049+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:31.074 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:31.068+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:31.084 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:31.080+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:31.165 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:31.159+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:31.416 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:31.414+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:31.758 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:31.754+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:31.760 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:31.757+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 10 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:32.049 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:32.045+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:32.064 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:32.057+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:32.084 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:32.076+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:32.155 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:32.149+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:32.389 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:32.385+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:32.776 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:32.773+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:32.779 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:32.778+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:33.012 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:33.009+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:33.063 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:33.057+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:33.081 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:33.075+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:33.202 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:33.197+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:33.406 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:33.404+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:33.761 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:33.757+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:33.762 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:33.760+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:33.975 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:33.973+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:34.029 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:34.013+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:34.081 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:34.063+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:34.246 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:34.240+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:34.396 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:34.393+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:34.779 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:34.777+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:34.781 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:34.780+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 11 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:34.974 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:34.971+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:35.018 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:35.012+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:35.081 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:35.075+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:35.203 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:35.197+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:35.443 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:35.440+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:35.786 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:35.782+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:35.787 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:35.785+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:35.941 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:35.938+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:36.034 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:36.028+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:36.098 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:36.091+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:36.255 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:36.250+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:36.409 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:36.406+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:36.758 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:36.755+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:36.759 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:36.757+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 12 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:36.950 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:36.948+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:37.011 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:37.006+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:37.149 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:37.143+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:37.217 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:37.211+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:37.447 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:37.443+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:37.810 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:37.807+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:37.814 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:37.812+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:37.979 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:37.973+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:37.999 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:37.996+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:38.179 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:38.172+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:38.205 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:38.199+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:38.410 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:38.406+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:38.853 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:38.849+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:38.855 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:38.852+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:38.985 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:38.976+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:39.038 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:39.035+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:39.150 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:39.140+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:39.200 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:39.193+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:39.446 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:39.443+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:39.808 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:39.806+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:39.810 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:39.809+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 13 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:40.002 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:39.996+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:40.029 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:40.027+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:40.161 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:40.152+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:40.216 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:40.208+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:40.413 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:40.412+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:40.858 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:40.854+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:40.858 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:40.856+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:41.015 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:41.013+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:41.046 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:41.040+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:41.149 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:41.144+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:41.175 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:41.169+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:41.368 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:41.365+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:41.800 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:41.788+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:41.897 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:41.893+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:41.898 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:41.896+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:41.997 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:41.994+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:42.037 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:42.031+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:42.134 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:42.127+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:42.223 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:42.217+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:42.375 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:42.372+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:42.807 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:42.801+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:42.870 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:42.864+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:42.871 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:42.868+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:43.025 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:43.016+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:43.067 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:43.021+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:43.185 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:43.180+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:43.247 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:43.241+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:43.414 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:43.410+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:43.773 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:43.767+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:43.904 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:43.898+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:43.905 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:43.901+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:43.978 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:43.972+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:43.989 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:43.987+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:44.210 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:44.204+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:44.245 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:44.228+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:44.425 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:44.421+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:44.750 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:44.744+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:44.935 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:44.931+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:44.937 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:44.934+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:44.951 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:44.946+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:44.966 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:44.962+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:45.185 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:45.177+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:45.256 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:45.251+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:45.415 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:45.410+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:45.716 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:45.710+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:45.853 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:21:45.851+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e278 v278) 2019-12-10T02:21:45.958 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:45.955+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:45.960 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:45.957+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:45.989 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:45.978+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:46.016 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:46.013+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:46.189 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:46.182+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:46.231 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:46.225+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:46.456 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:46.452+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:46.680 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:46.672+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:47.000 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:46.997+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:47.004 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:47.001+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:47.005 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:46.999+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:47.064 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:47.059+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:47.208 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:47.201+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:47.225 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:47.220+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:47.503 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:47.500+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:47.668 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:47.662+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:47.983 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:47.977+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:47.989 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:47.983+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:47.990 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:47.987+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:48.077 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:48.074+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:48.233 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:48.228+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:48.259 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:48.253+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:48.534 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:48.528+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:48.639 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:48.631+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:48.998 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:48.993+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:49.000 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:48.996+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:49.006 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:48.989+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:49.066 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:49.062+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:49.264 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:49.257+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:49.296 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:49.289+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:49.515 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:49.510+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:49.652 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:49.646+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:50.009 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:50.002+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:50.039 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:50.035+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:50.044 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:50.037+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:50.110 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:50.105+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:50.272 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:50.265+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:50.342 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:50.335+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:50.505 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:50.502+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:50.707 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:50.700+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:50.865 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:21:50.863+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e278 v278) 2019-12-10T02:21:51.006 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:51.001+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:51.008 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:51.004+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:51.012 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:51.008+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 15 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:51.104 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:51.099+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:51.275 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:51.268+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:51.303 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:51.295+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:51.552 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:51.547+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:51.695 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:51.688+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:52.006 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:52.003+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:52.011 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:52.008+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:52.018 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:52.011+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:52.109 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:52.103+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:52.272 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:52.265+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:52.341 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:52.333+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:52.521 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:52.517+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:52.683 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:52.677+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:53.036 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:53.032+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:53.043 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:53.038+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 16 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:53.056 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:53.050+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:53.156 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:53.152+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:53.261 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:53.254+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:53.295 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:53.288+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:53.552 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:53.549+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:53.716 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:53.711+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:54.067 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:54.061+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:54.097 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:54.094+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:54.098 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:54.096+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 17 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:54.185 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:54.180+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:54.305 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:54.299+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:54.315 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:54.309+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:54.592 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:54.588+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:54.683 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:54.677+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:55.070 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:55.061+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:55.152 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:55.145+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:55.153 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:55.149+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:55.218 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:55.214+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:55.316 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:55.310+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:55.321 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:55.312+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:55.591 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:55.587+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:55.707 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:55.701+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:55.888 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:21:55.882+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e278 v278) 2019-12-10T02:21:56.068 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:56.058+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:56.107 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:56.102+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:56.109 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:56.105+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:56.236 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:56.228+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:56.285 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:56.279+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:56.369 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:56.359+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:56.586 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:56.581+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:56.752 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:56.747+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:57.043 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:57.038+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:57.092 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:57.089+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:57.095 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:57.092+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:57.212 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:57.208+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:57.276 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:57.270+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:57.381 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:57.374+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:57.640 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:57.635+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:57.757 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:57.751+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:58.016 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:58.010+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:58.114 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:58.110+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:58.119 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:58.116+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:58.225 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:58.219+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:58.303 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:58.298+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:58.414 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:58.410+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:58.613 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:58.610+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:58.765 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:58.759+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:21:59.029 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:21:59.022+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:21:59.123 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:59.120+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:21:59.136 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:21:59.124+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:21:59.236 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:21:59.233+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:21:59.339 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:21:59.333+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:21:59.456 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:21:59.450+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:21:59.660 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:21:59.657+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:21:59.754 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:21:59.749+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:22:00.061 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:22:00.056+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:22:00.166 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:22:00.163+0000 1694f700 -1 osd.3 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:20:47.058360+0000 (oldest deadline 2019-12-10T02:21:27.058360+0000) 2019-12-10T02:22:00.168 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:22:00.166+0000 1694f700 -1 osd.3 278 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:22:00.251 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:22:00.248+0000 1694f700 -1 osd.2 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:27.320087+0000 front 2019-12-10T02:20:27.445822+0000 (oldest deadline 2019-12-10T02:21:13.009153+0000) 2019-12-10T02:22:00.357 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:22:00.353+0000 1694f700 -1 osd.4 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.222224+0000 front 2019-12-10T02:20:24.303404+0000 (oldest deadline 2019-12-10T02:21:09.233268+0000) 2019-12-10T02:22:00.421 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:22:00.415+0000 1694f700 -1 osd.7 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.559825+0000 front 2019-12-10T02:20:24.385709+0000 (oldest deadline 2019-12-10T02:21:09.638476+0000) 2019-12-10T02:22:00.635 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:22:00.632+0000 1694f700 -1 osd.0 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:25.917040+0000 front 2019-12-10T02:20:26.028714+0000 (oldest deadline 2019-12-10T02:21:11.783572+0000) 2019-12-10T02:22:00.739 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:22:00.734+0000 1694f700 -1 osd.5 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:24.288092+0000 front 2019-12-10T02:20:24.555714+0000 (oldest deadline 2019-12-10T02:21:41.211255+0000) 2019-12-10T02:22:00.943 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:22:00.936+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e278 v278) 2019-12-10T02:22:01.074 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:22:01.070+0000 1694f700 -1 osd.6 278 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:20:26.056645+0000 front 2019-12-10T02:20:26.083026+0000 (oldest deadline 2019-12-10T02:21:11.931026+0000) 2019-12-10T02:22:01.513 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:22:01.204+0000 1694f700 -1 osd.3 279 get_health_metrics reporting 19 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e276) 2019-12-10T02:22:38.871 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:22:38.868+0000 36f91700 -1 osd.2 283 build_incremental_map_msg missing incremental map 265 2019-12-10T02:22:39.589 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:22:39.576+0000 33f8b700 -1 osd.1 283 build_incremental_map_msg missing incremental map 260 2019-12-10T02:23:03.564 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:03.550+0000 35f8f700 -1 osd.5 283 build_incremental_map_msg missing incremental map 265 2019-12-10T02:23:31.501 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:49:52.562 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:23:31.537 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:49:52.600 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:23:31.568 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:49:52.632 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:23:31.575 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:49:52.639 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:23:31.605 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:49:52.764 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:23:31.635 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:49:52.791 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:23:31.732 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:49:52.793 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:23:31.754 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:49:52.816 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:23:31.791 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:49:52.950 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:23:33.274 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:33.271+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:33.496 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:33.494+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:34.022 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:34.010+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:34.289 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:34.286+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:34.505 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:34.502+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:34.977 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:34.971+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:35.163 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:35.156+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:35.293 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:35.288+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:35.477 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:35.473+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:36.025 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:36.020+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:36.029 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:36.025+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:36.181 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:36.176+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:36.216 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:36.210+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:36.321 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:36.314+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:36.430 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:36.427+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:36.914 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:36.911+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:37.029 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:37.023+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:37.031 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:37.027+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:37.167 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:37.161+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:37.230 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:37.222+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:37.313 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:37.307+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:37.401 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:37.397+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:37.961 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:37.954+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:38.051 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:38.040+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:38.051 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:38.043+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:38.195 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:38.180+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:38.211 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:38.190+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:38.354 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:38.338+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:38.385 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:38.381+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:38.942 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:38.939+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:39.051 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:39.046+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:39.054 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:39.050+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:39.165 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:39.159+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:39.209 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:39.203+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:39.354 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:39.351+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:39.429 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:39.426+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:39.922 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:39.919+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:40.086 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:40.081+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:40.089 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:40.083+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:40.155 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:40.150+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:40.192 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:40.187+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:40.317 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:40.308+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:40.421 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:40.416+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:40.948 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:40.945+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:41.088 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:41.082+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:41.090 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:41.084+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:41.169 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:41.163+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:41.237 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:41.228+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:41.341 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:41.336+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:41.470 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:41.466+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:41.964 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:41.960+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:42.115 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:42.109+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:42.116 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:42.111+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:42.126 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:42.120+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:42.203 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:42.197+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:42.386 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:42.380+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:42.504 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:42.501+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:42.988 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:42.985+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:43.078 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:43.072+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:43.079 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:43.074+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:43.180 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:43.172+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:43.182 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:43.176+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:43.396 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:43.393+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:43.507 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:43.505+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:43.997 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:43.995+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:44.059 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:44.053+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:44.061 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:44.058+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:44.187 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:44.181+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:44.229 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:44.216+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:44.419 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:44.418+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:44.545 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:44.543+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:45.033 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:45.030+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:45.098 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:45.092+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:45.101 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:45.097+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:45.170 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:45.164+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:45.259 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:45.254+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:45.403 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:45.400+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:45.529 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:45.527+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:46.012 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:46.010+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:46.090 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:46.084+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:46.091 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:46.086+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:46.190 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:46.182+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:46.233 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:46.228+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:46.423 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:46.420+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:46.494 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:46.491+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:46.997 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:46.995+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:47.103 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:47.093+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:47.103 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:47.095+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:47.189 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:47.183+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:47.214 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:47.207+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:47.441 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:47.438+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:47.466 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:47.463+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:47.968 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:47.966+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:48.069 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:48.064+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:48.072 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:48.068+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:48.177 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:48.171+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:48.202 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:48.197+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:48.484 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:48.479+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:48.504 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:48.502+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:48.987 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:48.984+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:49.080 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:49.074+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:49.081 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:49.076+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:49.183 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:49.171+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:49.238 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:49.233+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:49.463 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:49.459+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:49.541 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:49.538+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:49.955 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:49.951+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:50.061 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:50.052+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:50.062 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:50.053+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:50.145 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:50.141+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:50.282 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:50.276+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:50.348 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:50.341+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:50.490 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:50.488+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:50.508 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:50.505+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:50.963 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:50.960+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:51.060 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:51.052+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:51.061 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:51.056+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:51.179 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:51.174+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:51.285 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:51.279+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:51.339 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:51.332+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:51.454 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:51.452+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:51.467 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:51.463+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:51.943 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:51.939+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:52.088 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:52.082+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:52.088 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:52.084+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:52.164 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:52.157+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:52.249 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:52.237+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:52.345 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:52.340+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:52.419 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:52.418+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:52.485 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:52.482+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:52.978 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:52.975+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:53.111 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:53.104+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:53.111 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:53.107+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:53.221 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:53.208+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:53.286 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:53.281+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:53.398 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:53.392+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:53.469 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:53.466+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:53.470 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:53.468+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:54.004 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:54.000+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:54.128 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:54.117+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:54.129 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:54.120+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:54.201 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:54.196+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:54.255 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:54.241+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:54.399 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:54.393+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:54.471 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:54.469+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:54.490 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:54.489+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:55.035 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:55.033+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:55.092 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:55.086+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:55.093 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:55.088+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:55.225 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:55.217+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:55.239 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:55.233+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:55.403 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:55.398+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:55.488 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:55.486+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:55.490 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:55.489+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:55.992 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:55.990+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:56.140 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:56.135+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:56.142 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:56.138+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:56.203 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:56.198+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:56.273 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:56.261+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:56.428 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:56.422+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:56.455 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:56.447+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:56.510 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:56.508+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:56.957 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:56.955+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:57.107 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:57.102+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:57.110 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:57.106+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:57.196 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:57.189+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:57.225 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:57.219+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:57.446 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:57.440+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:57.493 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:57.490+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:57.512 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:57.510+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:57.941 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:57.938+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:58.104 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:58.099+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:58.107 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:58.104+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:58.171 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:58.165+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:58.264 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:58.258+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:58.427 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:58.421+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:58.447 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:58.444+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:58.512 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:58.510+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:58.905 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:58.900+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:23:59.121 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:59.115+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:23:59.124 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:23:59.119+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:23:59.175 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:23:59.169+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:23:59.242 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:23:59.236+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:23:59.437 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:23:59.433+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:23:59.441 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:23:59.436+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:23:59.504 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:23:59.499+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:23:59.957 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:23:59.952+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:00.139 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:00.133+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:00.142 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:00.136+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:00.143 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:00.138+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:00.261 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:00.256+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:00.402 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:00.397+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:00.473 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:00.470+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:00.483 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:00.478+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:00.960 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:00.956+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:01.143 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:01.129+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:01.183 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:01.178+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:01.187 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:01.183+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:01.231 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:01.223+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:01.440 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:01.436+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:01.490 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:01.483+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:01.491 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:01.488+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:01.960 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:01.956+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:02.122 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:02.116+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:02.218 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:02.211+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:02.241 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:02.234+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:02.241 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:02.236+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:02.404 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:02.399+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:02.466 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:02.462+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:02.470 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:02.465+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:02.913 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:02.910+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:03.098 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:03.089+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:03.230 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:03.224+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:03.233 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:03.228+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:03.267 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:03.261+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:03.410 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:03.405+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:03.505 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:03.501+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:03.515 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:03.509+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:03.967 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:03.963+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:04.153 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:04.144+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:04.263 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:04.256+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:04.265 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:04.260+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:04.316 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:04.309+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:04.415 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:04.411+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:04.528 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:04.524+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:04.540 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:04.534+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:05.011 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:05.006+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:05.128 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:05.122+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:05.302 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:05.296+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:05.303 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:05.297+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:05.303 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:05.299+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:05.385 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:05.381+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:05.544 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:05.539+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:05.581 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:05.575+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:05.969 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:05.964+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:06.098 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:06.092+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:06.317 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:06.312+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:06.319 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:06.315+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:06.339 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:06.334+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:06.367 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:06.364+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:06.507 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:06.504+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:06.554 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:06.549+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:06.575 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:24:06.570+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 44sec e283 v283) 2019-12-10T02:24:06.977 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:06.973+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:07.069 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:07.064+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:07.300 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:07.293+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:07.307 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:07.299+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:07.307 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:07.300+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:07.409 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:07.406+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:07.520 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:07.514+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:07.534 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:07.531+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:07.962 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:07.958+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:08.128 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:08.119+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:08.273 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:08.267+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:08.346 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:08.341+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:08.349 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:08.345+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:08.379 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:08.377+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:08.486 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:08.475+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:08.512 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:08.507+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:08.974 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:08.966+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:09.107 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:09.101+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:09.314 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:09.309+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:09.377 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:09.371+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:09.380 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:09.376+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:09.387 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:09.384+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:09.516 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:09.511+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:09.527 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:09.523+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:09.959 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:09.955+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:10.104 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:10.096+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:10.284 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:10.278+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:10.339 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:10.332+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:10.343 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:10.337+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:10.410 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:10.407+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:10.570 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:10.565+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:10.571 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:10.566+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:10.920 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:10.916+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:11.124 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:11.118+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:11.252 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:11.247+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:11.337 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:11.331+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:11.337 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:11.333+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:11.423 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:11.419+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:11.557 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:11.544+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:11.586 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:24:11.584+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 44sec e283 v283) 2019-12-10T02:24:11.612 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:11.609+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:11.921 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:11.915+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:12.158 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:12.152+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:12.251 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:12.243+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:12.344 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:12.338+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:12.345 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:12.340+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:12.467 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:12.464+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:12.523 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:12.516+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:12.590 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:12.588+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:12.939 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:12.936+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:13.142 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:13.136+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:13.281 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:13.275+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:13.340 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:13.334+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:13.341 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:13.336+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:13.505 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:13.501+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:13.526 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:13.518+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:13.571 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:13.567+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:13.906 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:13.902+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:14.149 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:14.143+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:14.283 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:14.277+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:14.313 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:14.304+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:14.313 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:14.306+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:14.470 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:14.466+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:14.523 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:14.518+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:14.620 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:14.615+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:14.891 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:14.888+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:15.131 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:15.124+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:15.273 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:15.264+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:15.333 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:15.324+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:15.334 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:15.327+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:15.464 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:15.461+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:15.494 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:15.488+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:15.623 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:15.620+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:15.885 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:15.880+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:16.177 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:24:16.169+0000 1694f700 -1 osd.4 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.153835+0000 front 2019-12-10T02:22:51.156875+0000 (oldest deadline 2019-12-10T02:23:34.424364+0000) 2019-12-10T02:24:16.273 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:24:16.267+0000 1694f700 -1 osd.6 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:50.656508+0000 front 2019-12-10T02:22:50.701545+0000 (oldest deadline 2019-12-10T02:23:35.913339+0000) 2019-12-10T02:24:16.373 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:16.367+0000 1694f700 -1 osd.5 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:49.395523+0000 front 2019-12-10T02:22:49.298365+0000 (oldest deadline 2019-12-10T02:23:33.925723+0000) 2019-12-10T02:24:16.374 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:16.369+0000 1694f700 -1 osd.5 283 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:16.441 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:24:16.437+0000 1694f700 -1 osd.2 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:52.963528+0000 front 2019-12-10T02:22:47.015352+0000 (oldest deadline 2019-12-10T02:23:32.821811+0000) 2019-12-10T02:24:16.510 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:24:16.504+0000 1694f700 -1 osd.7 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.169850+0000 front 2019-12-10T02:22:51.191470+0000 (oldest deadline 2019-12-10T02:23:49.981847+0000) 2019-12-10T02:24:16.630 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:24:16.626+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 44sec e283 v283) 2019-12-10T02:24:16.633 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:24:16.630+0000 1694f700 -1 osd.1 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:51.158940+0000 front 2019-12-10T02:22:50.450082+0000 (oldest deadline 2019-12-10T02:23:33.311913+0000) 2019-12-10T02:24:16.920 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:24:16.908+0000 1694f700 -1 osd.0 283 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:22:53.105128+0000 front 2019-12-10T02:22:52.722117+0000 (oldest deadline 2019-12-10T02:23:36.528891+0000) 2019-12-10T02:24:17.626 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:17.607+0000 1694f700 -1 osd.5 284 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:18.646 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:18.594+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:19.605 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:19.601+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:20.635 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:20.630+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:21.629 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:21.624+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:22.595 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:22.590+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:23.567 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:23.561+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:24.535 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:24.532+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:25.587 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:25.582+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:26.586 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:26.582+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:27.558 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:27.553+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:28.597 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:28.593+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:29.613 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:29.603+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:30.636 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:30.631+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:31.661 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:31.657+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:32.707 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:32.702+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:33.664 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:33.659+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:34.638 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:34.634+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:35.649 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:35.645+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:36.700 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:36.696+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:37.726 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:37.721+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:38.763 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:38.759+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:39.748 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:39.740+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:40.775 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:40.772+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:41.732 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:41.728+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:42.756 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:42.751+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:43.791 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:43.787+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:44.827 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:44.823+0000 1694f700 -1 osd.5 285 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:46.053 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:46.016+0000 1694f700 -1 osd.5 286 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:46.856 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:51:07.919 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:24:46.865 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:51:07.929 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:24:46.875 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:51:07.938 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:24:46.879 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:51:07.942 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:24:46.879 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:51:08.038 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:24:46.888 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:51:08.047 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:24:46.975 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:46.971+0000 1694f700 -1 osd.5 286 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:46.984 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:51:08.047 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:24:46.999 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:51:08.062 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:24:47.039 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:51:08.198 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:24:47.953 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:47.949+0000 1694f700 -1 osd.5 286 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:49.069 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:48.989+0000 1694f700 -1 osd.5 287 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:50.044 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:50.034+0000 1694f700 -1 osd.5 288 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:50.641 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:24:50.605+0000 31f87700 -1 osd.3 288 build_incremental_map_msg missing incremental map 288 2019-12-10T02:24:50.940 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:24:50.932+0000 30f85700 -1 osd.3 288 build_incremental_map_msg missing incremental map 288 2019-12-10T02:24:51.025 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:51.021+0000 1694f700 -1 osd.5 289 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:51.049 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:24:50.954+0000 35f8f700 -1 osd.3 288 build_incremental_map_msg missing incremental map 288 2019-12-10T02:24:52.049 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:52.045+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:52.065 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:24:52.060+0000 34f8d700 -1 osd.3 289 build_incremental_map_msg missing incremental map 288 2019-12-10T02:24:53.028 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:53.012+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:54.012 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:54.008+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:55.203 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:55.139+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:56.144 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:56.141+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:57.183 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:57.178+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:58.147 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:58.142+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:24:59.112 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:24:59.107+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:00.150 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:00.146+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:01.132 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:01.128+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:02.152 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:02.147+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:03.141 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:03.136+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:04.172 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:04.168+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:05.170 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:05.166+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:06.159 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:06.155+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:07.161 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:07.140+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:08.158 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:08.143+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:09.140 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:09.136+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:10.099 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:10.095+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:11.070 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:11.065+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:12.102 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:12.097+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:13.136 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:13.130+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:14.096 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:14.092+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:14.909 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:14.903+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:14.910 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:14.903+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:15.104 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:15.097+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:15.108 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:15.104+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:15.466 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:15.465+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:15.881 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:15.875+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:15.891 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:15.885+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:16.102 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:16.096+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:16.103 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:16.099+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:16.139 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:16.137+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:16.511 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:16.508+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:16.866 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:16.854+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:16.922 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:16.916+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:17.099 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:17.096+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:17.152 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:17.145+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:17.159 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:17.153+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:17.470 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:17.466+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:17.859 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:17.850+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:17.884 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:17.878+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:18.079 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:18.076+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:18.154 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:18.148+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:18.155 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:18.151+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:18.521 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:18.518+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:18.811 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:18.806+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:18.846 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:18.840+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:19.045 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:19.042+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:19.205 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:19.200+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:19.209 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:19.205+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:19.542 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:19.539+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:19.769 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:19.764+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:19.805 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:19.799+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:19.824 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:19.817+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:19.981 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:19.978+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:20.088 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:20.085+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:20.195 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:20.189+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:20.196 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:20.192+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:20.535 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:20.533+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:20.782 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:20.776+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:20.785 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:20.779+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:20.796 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:20.791+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:21.025 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:21.024+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:21.075 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:21.072+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:21.163 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:21.157+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:21.164 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:21.159+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:21.516 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:21.513+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:21.764 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:21.759+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:21.770 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:21.765+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:21.775 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:21.769+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:22.044 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:22.042+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:22.054 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:22.054+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 14 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:22.161 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:22.154+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:22.161 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:22.157+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:22.525 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:22.522+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:22.757 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:22.752+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:22.788 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:22.784+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:22.799 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:22.793+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:23.030 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:23.028+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:23.043 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:23.041+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:23.128 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:23.121+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:23.129 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:23.124+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:23.485 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:23.482+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:23.763 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:23.757+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:23.809 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:23.803+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:23.830 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:23.823+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:23.999 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:23.995+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:24.010 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:24.008+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:24.147 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:24.141+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:24.147 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:24.143+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:24.468 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:24.465+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:24.719 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:24.714+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:24.767 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:24.762+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:24.860 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:24.855+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:24.968 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:24.966+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:24.972 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:24.970+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:25.196 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:25.191+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:25.196 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:25.193+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:25.497 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:25.493+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:25.753 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:25.746+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:25.767 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:25.761+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:25.881 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:25.874+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:25.935 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:25.932+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:25.988 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:25.986+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:26.169 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:26.162+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:26.169 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:26.165+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:26.488 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:26.482+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:26.728 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:26.721+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:26.808 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:26.802+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:26.868 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:26.861+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:26.952 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:26.947+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:27.019 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:27.016+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:27.021 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:27.018+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:27.179 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:27.174+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:27.180 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:27.176+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:27.479 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:27.476+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:27.752 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:27.746+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:27.810 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:27.805+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:27.871 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:27.865+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:27.908 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:27.904+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:27.991 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:27.988+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:27.995 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:27.992+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:28.217 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:28.211+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:28.218 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:28.213+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:28.463 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:28.459+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:28.747 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:28.741+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:28.779 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:28.774+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:28.860 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:28.857+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:28.887 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:28.880+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:28.976 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:28.973+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:28.981 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:28.977+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:29.252 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:29.246+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:29.253 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:29.248+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:29.495 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:29.491+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:29.719 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:29.712+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:29.758 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:29.752+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:29.891 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:29.884+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:29.900 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:29.898+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:29.978 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:29.970+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:29.983 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:29.976+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:30.211 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:30.207+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:30.212 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:30.209+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:30.467 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:30.464+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:30.724 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:30.719+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:30.728 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:30.722+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:30.910 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:30.903+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:30.916 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:30.912+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:31.015 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:31.013+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:31.019 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:31.015+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:31.245 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:31.238+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:31.245 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:31.241+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:31.477 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:31.473+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:31.689 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:31.684+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:31.752 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:31.746+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:31.937 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:31.928+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:31.950 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:31.946+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:32.042 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:32.037+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:32.043 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:32.040+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:32.262 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:32.257+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:32.265 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:32.261+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:32.518 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:32.515+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:32.647 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:32.641+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:32.716 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:32.710+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:32.910 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:32.904+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:32.974 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:32.970+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:33.085 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:33.082+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:33.088 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:33.086+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:33.305 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:33.299+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:33.306 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:33.301+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:33.475 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:33.470+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:33.661 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:33.653+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:33.683 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:33.677+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:33.932 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:33.925+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:33.976 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:33.971+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:34.050 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:34.046+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:34.059 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:34.051+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:34.267 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:34.261+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:34.267 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:34.263+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:34.515 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:34.512+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:34.700 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:34.694+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:34.705 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:34.699+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:34.960 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:34.954+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:34.997 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:34.993+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:35.105 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:35.099+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:35.107 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:35.103+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:35.310 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:35.304+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:35.313 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:35.310+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:35.564 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:35.560+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:35.662 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:35.656+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:35.688 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:35.682+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:35.969 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:35.964+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:36.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:36.049+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:36.083 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:36.075+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:36.084 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:36.081+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:36.323 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:36.315+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:36.327 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:36.323+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:36.595 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:36.592+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:36.694 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:36.688+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:36.730 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:36.724+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:36.988 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:36.982+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:37.028 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:37.023+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:37.092 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:37.089+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:37.096 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:37.093+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:37.348 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:37.341+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:37.348 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:37.344+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:37.589 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:37.583+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:37.662 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:37.656+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:37.767 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:37.761+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:37.992 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:37.987+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:38.030 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:38.024+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:38.087 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:38.080+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:38.088 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:38.084+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:38.382 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:38.377+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:38.385 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:38.381+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:38.555 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:38.551+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:38.635 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:38.629+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:38.740 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:38.735+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:38.971 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:38.965+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:39.075 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:39.070+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:39.076 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:39.073+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:39.084 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:39.077+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:39.373 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:39.361+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:39.373 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:39.366+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:39.598 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:39.593+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:39.639 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:39.633+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:39.780 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:39.773+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:39.941 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:39.936+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:40.122 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:40.115+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:40.122 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:40.118+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:40.127 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:40.119+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:40.339 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:40.333+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:40.340 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:40.335+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:40.600 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:40.598+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:40.613 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:40.608+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:40.758 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:40.753+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:40.909 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:40.904+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:41.085 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:41.076+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:41.169 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:41.163+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:41.170 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:41.166+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:41.364 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:41.358+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:41.365 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:41.360+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:41.580 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:41.572+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:41.664 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:41.658+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:41.723 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:41.716+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:41.866 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:41.861+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:42.060 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:52:03.123 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:25:42.104 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:42.099+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:42.113 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:52:03.176 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:25:42.123 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:52:03.186 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:25:42.129 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:52:03.193 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:25:42.134 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:52:03.294 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:25:42.139 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:52:03.299 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:25:42.200 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:52:03.263 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:25:42.220 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:42.216+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:42.221 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:52:03.284 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:25:42.224 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:42.220+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:42.241 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:52:03.400 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:25:42.366 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:42.360+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:42.367 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:42.363+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:42.629 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:42.625+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:42.695 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:42.691+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:42.697 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:42.692+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:42.900 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:42.896+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:43.156 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:43.151+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:43.236 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:43.233+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:43.240 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:43.237+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:43.328 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:43.323+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:43.328 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:43.324+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:43.597 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:43.594+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:43.685 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:43.679+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:43.732 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:43.726+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:43.865 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:43.861+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:44.187 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:44.182+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:44.200 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:44.196+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:44.202 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:44.199+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:44.324 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:44.318+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:44.325 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:44.320+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:44.571 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:44.568+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:44.689 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:44.683+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:44.707 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:44.701+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:44.834 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:44.832+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:45.216 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:45.212+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:45.221 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:45.217+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:45.230 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:45.223+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:45.351 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:45.344+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:45.354 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:45.350+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:45.554 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:45.550+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:45.717 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:45.712+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:45.730 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:45.724+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:45.871 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:45.859+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:46.185 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:46.178+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:46.197 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:46.182+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:46.267 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:46.262+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:46.362 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:46.355+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:46.363 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:46.358+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:46.554 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:46.551+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:46.738 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:46.732+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:46.769 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:46.762+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:46.875 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:46.854+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:47.026 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:25:47.024+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e290 v290) 2019-12-10T02:25:47.169 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:47.160+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:47.171 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:47.164+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:47.318 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:47.310+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:47.408 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:47.402+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:47.412 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:47.408+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:47.556 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:47.552+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:47.775 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:47.768+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:47.783 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:47.779+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:47.882 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:47.877+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:48.141 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:48.132+0000 1694f700 -1 osd.3 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:24:46.475439+0000 (oldest deadline 2019-12-10T02:25:26.475439+0000) 2019-12-10T02:25:48.141 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:48.136+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:48.294 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:48.288+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:48.377 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:48.371+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:48.378 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:48.374+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:48.600 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:48.597+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:48.792 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:48.786+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:48.812 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:48.806+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:48.844 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:48.841+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:49.105 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:49.102+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:49.306 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:49.299+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:49.367 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:49.361+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:49.367 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:49.363+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:49.645 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:49.642+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:49.764 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:49.758+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:49.846 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:49.843+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:49.865 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:49.859+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:50.143 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:50.139+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:50.291 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:50.285+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:50.391 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:50.385+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:50.392 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:50.387+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:50.670 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:50.667+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:50.745 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:50.739+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:50.809 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:50.806+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:50.852 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:50.846+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:51.155 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:51.152+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:51.323 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:51.317+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:51.434 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:51.428+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:51.437 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:51.433+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:51.701 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:51.697+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:51.763 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:51.754+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:51.821 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:51.817+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:51.896 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:51.891+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:52.050 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:25:52.044+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e290 v290) 2019-12-10T02:25:52.134 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:52.132+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:52.284 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:52.278+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:52.484 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:52.477+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:52.485 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:52.480+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:52.696 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:52.692+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:52.729 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:52.724+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:52.807 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:52.804+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:52.940 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:52.932+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:53.179 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:53.175+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:53.302 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:53.296+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:53.472 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:53.465+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:53.472 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:53.468+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:53.720 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:53.715+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:53.765 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:53.758+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:53.854 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:53.849+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:53.938 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:53.932+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:54.144 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:54.138+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:54.315 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:54.309+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:54.525 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:54.520+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:54.527 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:54.524+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:54.697 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:54.693+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:54.793 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:54.786+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:54.899 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:54.895+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:54.947 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:54.941+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:55.193 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:55.189+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:55.288 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:55.282+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:55.557 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:55.551+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:55.560 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:55.556+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:55.684 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:55.680+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:55.785 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:55.779+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:55.943 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:55.939+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:55.957 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:55.951+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:56.238 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:56.235+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:56.274 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:56.269+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:56.587 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:56.583+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:56.588 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:56.584+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:56.660 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:56.657+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:56.770 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:56.763+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:56.925 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:56.921+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:56.965 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:56.960+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:57.061 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:25:57.057+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e290 v290) 2019-12-10T02:25:57.283 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:57.281+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:57.289 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:57.283+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:57.552 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:57.547+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:57.554 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:57.548+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:57.653 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:57.650+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:57.794 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:57.787+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:57.928 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:57.923+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:58.020 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:58.014+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:58.276 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:58.271+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:58.321 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:58.319+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:58.527 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:58.521+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:58.530 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:58.526+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:58.613 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:58.611+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:58.767 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:58.761+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:58.944 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:58.941+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:25:59.066 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:25:59.059+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:25:59.286 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:25:59.284+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:25:59.324 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:25:59.318+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:25:59.523 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:59.518+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:25:59.526 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:25:59.522+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:25:59.663 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:25:59.660+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:25:59.740 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:25:59.734+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:25:59.976 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:25:59.974+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:26:00.075 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:26:00.070+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:26:00.263 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:26:00.262+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:26:00.285 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:26:00.278+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:26:00.486 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:26:00.477+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:26:00.491 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:26:00.480+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:26:00.671 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:26:00.669+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:26:00.769 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:26:00.763+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:26:00.995 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:26:00.993+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:26:01.119 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:26:01.112+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:26:01.220 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:26:01.218+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:26:01.308 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:26:01.302+0000 1694f700 -1 osd.7 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.147412+0000 front 2019-12-10T02:24:33.099474+0000 (oldest deadline 2019-12-10T02:25:18.955107+0000) 2019-12-10T02:26:01.497 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:26:01.486+0000 1694f700 -1 osd.5 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:31.162106+0000 front 2019-12-10T02:24:30.962894+0000 (oldest deadline 2019-12-10T02:25:15.021016+0000) 2019-12-10T02:26:01.497 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:26:01.488+0000 1694f700 -1 osd.5 290 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:26:01.683 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:26:01.680+0000 1694f700 -1 osd.0 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:30.202988+0000 front 2019-12-10T02:24:30.207075+0000 (oldest deadline 2019-12-10T02:25:14.966377+0000) 2019-12-10T02:26:01.815 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:26:01.809+0000 1694f700 -1 osd.6 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:32.816707+0000 front 2019-12-10T02:24:32.694671+0000 (oldest deadline 2019-12-10T02:25:14.321904+0000) 2019-12-10T02:26:01.963 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:26:01.961+0000 1694f700 -1 osd.2 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:33.360363+0000 front 2019-12-10T02:24:33.513841+0000 (oldest deadline 2019-12-10T02:25:15.610811+0000) 2019-12-10T02:26:02.129 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:26:02.123+0000 1694f700 -1 osd.4 290 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:24:29.426277+0000 front 2019-12-10T02:24:29.444986+0000 (oldest deadline 2019-12-10T02:25:14.103216+0000) 2019-12-10T02:26:02.131 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:26:02.125+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e290 v290) 2019-12-10T02:26:02.210 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:26:02.206+0000 1694f700 -1 osd.3 290 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4557.0:106147 7.2 7.3dc8e7a2 (undecoded) ondisk+retry+write+known_if_redirected e287) 2019-12-10T02:26:02.580 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:26:02.521+0000 1694f700 -1 osd.5 291 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109135 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e283) 2019-12-10T02:26:47.331 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:26:47.315+0000 32788700 -1 osd.3 296 build_incremental_map_msg missing incremental map 288 2019-12-10T02:27:00.020 INFO:teuthology.orchestra.run.smithi067.stderr:testIfMatch (test.functional.tests.TestFileComparison) ... ok 2019-12-10T02:27:38.207 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:38.202+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:38.286 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:38.282+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:39.220 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:39.214+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:39.338 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:39.335+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:39.567 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:39.561+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:39.913 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:39.911+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:39.969 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:39.959+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:40.015 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:40.005+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:40.015 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:40.008+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:40.256 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:40.248+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:40.348 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:40.346+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:40.556 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:40.549+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:40.883 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:40.880+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:40.937 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:40.932+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:41.049 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:41.041+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:41.049 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:41.043+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:41.120 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:41.113+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:41.257 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:41.252+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:41.395 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:41.392+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:41.597 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:41.589+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:41.849 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:41.845+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:41.932 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:41.929+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:42.063 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:42.057+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:42.063 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:42.059+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:42.131 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:42.125+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:42.306 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:42.301+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:42.427 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:42.424+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:42.640 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:42.634+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:42.826 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:42.821+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:42.941 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:42.937+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:43.022 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:43.015+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:43.022 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:43.018+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:43.106 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:43.100+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:43.337 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:43.332+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:43.413 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:43.409+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:43.634 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:43.627+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:43.796 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:43.792+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:43.938 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:43.935+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:43.992 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:43.986+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:43.992 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:43.988+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:44.092 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:44.086+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:44.381 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:44.376+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:44.426 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:44.423+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:44.683 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:44.677+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:44.750 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:44.747+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:44.908 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:44.905+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:45.004 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:44.998+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:45.004 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:45.000+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:45.054 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:45.049+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:45.390 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:45.385+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:45.434 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:45.430+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:45.728 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:45.722+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:45.769 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:45.765+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:45.904 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:45.902+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:46.037 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:46.030+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:46.038 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:46.032+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:46.091 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:46.084+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:46.402 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:46.394+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:46.406 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:46.400+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:46.743 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:46.738+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:46.748 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:46.742+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:46.884 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:46.881+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:47.052 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:47.040+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:47.052 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:47.043+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:47.130 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:47.125+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:47.360 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:47.356+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:47.403 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:47.395+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:47.708 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:47.699+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:47.754 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:47.749+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:47.878 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:47.874+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:48.096 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:48.090+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:48.097 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:48.092+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:48.144 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:48.135+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:48.317 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:48.313+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:48.364 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:48.359+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:48.719 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:48.714+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:48.772 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:48.770+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:48.863 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:48.860+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:49.070 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:49.056+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:49.070 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:49.057+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:49.171 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:49.164+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:49.312 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:49.309+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:49.406 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:49.399+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:49.714 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:49.708+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:49.794 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:49.790+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:49.902 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:49.898+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:50.061 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:50.054+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:50.061 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:50.057+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:50.129 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:50.123+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:50.296 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:50.292+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:50.385 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:50.381+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:50.724 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:50.718+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:50.765 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:50.754+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:50.942 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:50.938+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:51.086 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:51.081+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:51.090 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:51.085+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:51.159 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:51.153+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:51.311 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:51.308+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:51.376 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:51.370+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:51.689 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:51.683+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:51.766 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:51.761+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:51.934 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:51.930+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:52.095 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:52.089+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:52.096 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:52.091+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:52.152 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:52.132+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:52.320 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:52.317+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:52.377 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:52.369+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:52.664 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:52.659+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:52.779 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:52.774+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:52.894 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:52.890+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:53.123 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:53.117+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:53.124 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:53.119+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:53.163 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:53.157+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:53.337 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:53.331+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:53.352 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:53.344+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:53.719 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:53.713+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:53.770 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:53.765+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:53.906 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:53.903+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:54.096 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:54.090+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:54.097 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:54.092+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:54.138 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:54.129+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:54.321 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:54.317+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:54.389 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:54.383+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:54.768 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:54.763+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:54.793 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:54.788+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:54.900 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:54.897+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:55.060 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:55.051+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:55.060 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:55.055+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:55.188 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:55.183+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:55.344 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:55.340+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:55.436 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:55.429+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:55.733 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:55.728+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:55.794 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:55.789+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:55.947 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:55.943+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:56.061 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:56.055+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:56.064 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:56.060+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:56.154 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:56.148+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:56.352 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:56.349+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:56.410 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:56.403+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:56.763 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:56.757+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:56.820 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:56.817+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:56.956 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:56.948+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:57.048 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:57.043+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:57.057 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:57.046+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:57.167 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:57.162+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:57.405 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:57.399+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:57.432 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:57.427+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:57.718 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:57.713+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:57.794 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:57.791+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:57.953 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:57.949+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:58.083 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:58.077+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:58.083 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:58.079+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:58.213 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:58.207+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:58.366 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:58.362+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:58.411 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:58.406+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:58.724 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:58.719+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:58.773 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:58.769+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:27:58.993 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:27:58.990+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:27:59.121 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:59.113+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:27:59.121 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:27:59.115+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:27:59.205 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:27:59.199+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:27:59.405 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:27:59.400+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:27:59.426 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:27:59.421+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:27:59.716 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:27:59.706+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:27:59.796 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:27:59.792+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:00.042 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:28:00.038+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:28:00.093 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:00.084+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:00.093 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:00.086+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:00.196 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:00.191+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:00.382 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:00.376+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:00.447 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:00.444+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:00.735 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:00.730+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:00.756 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:00.753+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:01.017 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:28:01.012+0000 1694f700 -1 osd.2 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.843088+0000 front 2019-12-10T02:26:56.946868+0000 (oldest deadline 2019-12-10T02:27:39.111617+0000) 2019-12-10T02:28:01.140 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:01.134+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:01.141 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:01.136+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:01.222 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:01.217+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:01.372 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:01.366+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:01.473 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:01.469+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:01.713 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:01.709+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:01.759 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:01.753+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:02.163 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:02.157+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:02.166 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:02.162+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:02.213 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:02.207+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:02.343 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:02.338+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:02.479 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:02.476+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:02.761 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:02.757+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:02.785 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:02.780+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:03.147 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:03.140+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:03.147 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:03.142+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:03.185 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:03.179+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:03.355 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:03.349+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:03.519 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:03.515+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:03.739 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:03.736+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:03.784 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:03.778+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:04.123 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:04.117+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:04.126 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:04.122+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:04.218 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:04.210+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:04.391 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:04.386+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:04.559 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:04.549+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:04.727 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:04.725+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:04.764 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:04.760+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:05.116 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:05.110+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:05.122 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:05.115+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:05.211 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:05.206+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:05.367 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:05.362+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:05.543 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:05.540+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:05.718 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:05.714+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:05.742 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:05.737+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:06.123 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:06.113+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:06.124 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:06.115+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:06.235 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:06.227+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:06.380 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:06.374+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:06.533 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:06.526+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:06.688 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:06.685+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:06.721 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:06.715+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:07.150 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:07.145+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:07.151 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:07.147+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:07.214 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:07.208+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:07.364 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:07.358+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:07.542 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:07.537+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:07.645 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:07.641+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:07.702 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:07.697+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:08.198 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:08.192+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:08.204 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:08.194+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:08.261 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:08.256+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:08.403 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:08.398+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:08.554 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:08.550+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:08.648 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:08.644+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:08.706 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:08.701+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:09.211 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:09.204+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:09.211 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:09.206+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:09.259 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:09.254+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:09.377 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:09.372+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:09.549 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:09.545+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:09.601 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:09.597+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:09.683 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:09.677+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:10.179 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:10.172+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:10.180 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:10.176+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:10.289 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:10.284+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:10.371 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:10.366+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:10.583 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:10.578+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:10.602 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:10.594+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:10.667 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:10.661+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:11.195 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:11.188+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:11.195 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:11.191+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:11.263 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:11.256+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:11.329 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:11.323+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:11.610 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:11.607+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:11.624 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:11.622+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:11.713 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:11.705+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:12.245 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:12.239+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:12.248 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:12.244+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:12.251 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:12.246+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:12.334 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:12.328+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:12.536 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:28:12.534+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 3 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 43sec e296 v296) 2019-12-10T02:28:12.644 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:12.643+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:12.650 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:12.648+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:12.745 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:12.740+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:13.224 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:13.217+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:13.224 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:13.219+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:13.225 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:13.219+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:13.321 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:13.317+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:13.611 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:13.605+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:13.681 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:13.678+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:13.773 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:13.767+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:14.218 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:14.211+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:14.218 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:14.212+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:14.218 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:14.214+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:14.361 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:14.356+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:14.584 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:14.581+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:14.665 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:14.663+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:14.751 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:14.746+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:15.245 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:15.239+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:15.248 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:15.244+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:15.249 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:15.244+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:15.383 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:15.377+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:15.617 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:15.614+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:15.658 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:15.657+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:15.752 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:15.746+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:16.284 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:16.277+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:16.284 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:16.280+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:16.293 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:16.288+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:16.407 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:16.401+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:16.619 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:16.617+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:16.631 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:16.629+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:16.712 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:16.707+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:17.315 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:17.310+0000 1694f700 -1 osd.5 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:56.584865+0000 front 2019-12-10T02:26:56.573924+0000 (oldest deadline 2019-12-10T02:27:39.996578+0000) 2019-12-10T02:28:17.318 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:17.314+0000 1694f700 -1 osd.5 296 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:17.339 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:28:17.334+0000 1694f700 -1 osd.7 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.732353+0000 front 2019-12-10T02:26:54.745554+0000 (oldest deadline 2019-12-10T02:27:40.590312+0000) 2019-12-10T02:28:17.373 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:28:17.367+0000 1694f700 -1 osd.4 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.254183+0000 front 2019-12-10T02:26:54.902835+0000 (oldest deadline 2019-12-10T02:27:37.771608+0000) 2019-12-10T02:28:17.611 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:28:17.609+0000 1694f700 -1 osd.1 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:55.411788+0000 front 2019-12-10T02:26:55.247731+0000 (oldest deadline 2019-12-10T02:27:38.036375+0000) 2019-12-10T02:28:17.620 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:28:17.617+0000 1694f700 -1 osd.0 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:54.778230+0000 front 2019-12-10T02:26:54.861804+0000 (oldest deadline 2019-12-10T02:27:39.411931+0000) 2019-12-10T02:28:17.630 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:28:17.624+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 43sec e296 v296) 2019-12-10T02:28:17.695 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:28:17.688+0000 1694f700 -1 osd.6 296 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:26:58.003605+0000 front 2019-12-10T02:26:58.049373+0000 (oldest deadline 2019-12-10T02:27:39.069790+0000) 2019-12-10T02:28:18.779 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:18.771+0000 1694f700 -1 osd.5 297 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:19.984 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:19.971+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:20.992 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:20.987+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:21.959 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:21.955+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:22.933 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:22.928+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:23.888 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:23.884+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:24.844 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:24.840+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:25.809 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:25.805+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:26.780 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:26.776+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:27.825 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:27.821+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:27.862 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:54:48.925 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:28:27.872 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:54:48.931 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:28:27.880 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:54:48.944 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:28:27.885 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:54:48.948 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:28:27.924 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:54:49.083 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:28:27.943 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:54:49.102 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:28:28.093 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:54:49.156 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:28:28.125 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:54:49.184 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:28:28.177 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:54:49.336 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:28:28.794 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:28.789+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:29.793 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:29.788+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:30.756 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:30.752+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:31.761 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:31.757+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:32.717 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:32.713+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:33.704 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:33.700+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:34.694 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:34.690+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:35.687 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:35.683+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:36.683 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:36.679+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:37.655 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:37.650+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:38.704 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:38.699+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:39.669 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:39.665+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:40.635 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:40.630+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:41.640 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:41.635+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:42.676 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:42.672+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:43.651 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:43.647+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:44.675 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:44.670+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:45.660 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:45.656+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:46.632 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:46.627+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:47.649 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:47.645+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:48.656 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:48.652+0000 1694f700 -1 osd.5 298 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:49.645 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:49.641+0000 1694f700 -1 osd.5 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:50.610 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:50.606+0000 1694f700 -1 osd.5 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:51.598 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:51.595+0000 1694f700 -1 osd.5 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:52.638 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:52.634+0000 1694f700 -1 osd.5 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:53.613 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:53.605+0000 1694f700 -1 osd.5 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:54.571 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:54.565+0000 1694f700 -1 osd.5 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:55.533 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:55.529+0000 1694f700 -1 osd.5 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:56.512 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:56.505+0000 1694f700 -1 osd.5 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:57.569 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:57.556+0000 1694f700 -1 osd.5 299 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:58.578 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:58.569+0000 1694f700 -1 osd.5 300 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:28:59.537 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:28:59.534+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:00.515 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:00.512+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:01.508 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:01.503+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:02.514 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:02.510+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:03.506 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:03.502+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:04.504 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:04.499+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:05.473 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:05.460+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:06.469 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:06.464+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:07.500 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:07.496+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:08.518 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:08.514+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:09.566 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:09.551+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:10.591 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:10.587+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:11.614 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:11.610+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:12.593 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:12.589+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:13.571 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:13.567+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:14.550 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:14.544+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:15.513 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:15.507+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:16.395 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:16.392+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:16.450 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:16.444+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:16.534 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:16.526+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:16.538 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:16.528+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:17.361 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:17.358+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:17.477 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:17.469+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:17.556 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:17.549+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:17.561 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:17.552+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:18.380 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:18.376+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:18.438 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:18.433+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:18.594 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:18.588+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:18.595 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:18.591+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:18.931 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:18.925+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:19.103 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:19.096+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:19.367 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:19.363+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:19.410 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:19.404+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:19.624 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:19.618+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:19.624 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:19.620+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:19.928 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:19.923+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:20.112 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:20.108+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:20.140 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:20.133+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:20.335 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:20.329+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:20.377 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:20.370+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:20.651 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:20.645+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:20.654 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:20.650+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:20.980 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:20.975+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:21.137 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:21.130+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:21.139 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:21.133+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:21.336 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:21.332+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:21.431 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:21.424+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:21.667 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:21.661+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:21.670 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:21.667+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:21.958 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:21.951+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:22.115 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:22.111+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:22.132 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:22.126+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:22.341 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:22.337+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:22.427 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:22.421+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:22.626 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:22.619+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:22.627 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:22.622+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:22.985 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:22.977+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:23.110 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:23.106+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:23.146 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:23.139+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:23.323 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:23.319+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:23.456 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:23.450+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:23.665 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:23.658+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:23.666 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:23.661+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:23.951 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:23.943+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:24.121 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:24.115+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:24.141 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:24.138+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:24.349 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:24.345+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:24.488 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:24.482+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:24.629 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:24.623+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:24.632 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:24.628+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:24.906 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:24.900+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:25.101 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:25.095+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:25.115 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:25.110+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:25.403 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:25.394+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:25.492 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:25.487+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:25.656 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:25.643+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:25.656 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:25.646+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:25.955 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:25.950+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:26.100 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:26.093+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:26.143 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:26.138+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:26.373 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:26.369+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:26.480 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:26.474+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:26.686 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:26.680+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:26.687 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:26.683+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:26.998 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:26.991+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:27.139 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:27.133+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:27.151 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:27.146+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:27.379 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:27.375+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:27.444 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:27.437+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:27.651 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:27.646+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:27.652 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:27.648+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:27.980 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:27.975+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:28.143 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:28.137+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:28.166 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:28.162+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:28.391 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:28.388+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:28.434 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:28.429+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:28.622 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:28.615+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:28.625 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:28.620+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:28.997 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:28.990+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:29.136 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:29.132+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:29.170 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:29.162+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:29.379 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:29.376+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:29.483 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:29.475+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:29.637 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:29.629+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:29.638 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:29.632+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:29.982 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:29.977+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:30.120 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:30.115+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:30.217 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:30.211+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:30.384 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:30.381+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:30.446 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:30.439+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:30.598 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:30.591+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:30.599 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:30.594+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:30.858 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:30.854+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:30.938 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:30.933+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:31.103 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:31.099+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:31.215 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:31.209+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:31.394 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:31.390+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:31.495 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:31.489+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:31.600 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:31.595+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:31.600 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:31.596+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:31.819 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:31.816+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:31.942 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:31.936+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:32.126 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:32.122+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:32.225 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:32.218+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:32.375 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:32.371+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:32.485 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:32.478+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:32.604 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:32.597+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:32.605 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:32.600+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:32.809 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:32.806+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:32.900 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:32.895+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:33.150 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:33.146+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:33.248 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:33.242+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:33.362 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:33.358+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:33.498 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:33.493+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:33.654 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:33.648+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:33.655 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:33.651+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:33.849 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:33.846+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:33.868 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:33.865+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:34.214 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:34.199+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:34.279 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:34.273+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:34.357 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:34.354+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:34.487 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:34.480+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:34.704 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:34.697+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:34.708 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:34.700+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:34.820 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:34.817+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:34.865 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:34.859+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:35.232 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:35.230+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:35.313 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:35.310+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:35.332 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:35.325+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:35.510 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:35.500+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:35.665 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:35.660+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:35.668 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:35.664+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:35.819 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:35.814+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:35.822 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:35.817+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:35.916 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:35.910+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:36.261 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:36.258+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:36.356 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:36.351+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:36.372 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:36.366+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:36.557 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:36.550+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:36.705 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:36.696+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:36.705 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:36.699+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:36.808 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:36.802+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:36.810 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:36.806+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:36.918 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:36.913+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:37.267 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:37.262+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:37.320 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:37.308+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:37.346 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:37.334+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:37.549 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:37.535+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:37.689 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:37.680+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:37.689 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:37.683+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:37.801 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:37.792+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:37.802 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:37.795+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:37.910 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:37.903+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:38.264 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:38.260+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:38.301 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:38.295+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:38.356 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:38.353+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:38.577 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:38.571+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:38.648 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:38.641+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:38.653 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:38.649+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:38.840 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:38.834+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:38.842 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:38.837+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:38.914 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:38.906+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:39.273 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:39.267+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:39.327 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:39.321+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:39.412 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:39.408+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:39.550 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:39.544+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:39.683 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:39.677+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:39.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:39.680+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:39.798 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:39.793+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:39.805 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:39.801+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:39.925 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:39.920+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:40.232 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:40.229+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:40.295 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:40.289+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:40.381 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:40.377+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:40.531 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:40.526+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:40.687 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:40.681+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:40.688 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:40.684+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:40.792 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:40.788+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:40.797 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:40.794+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:40.938 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:40.931+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:41.285 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:41.282+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:41.342 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:41.336+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:41.426 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:41.423+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:41.515 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:41.509+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:41.702 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:41.695+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:41.703 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:41.698+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:41.781 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:41.779+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:41.785 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:41.782+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:41.954 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:41.949+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:42.304 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:42.300+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:42.382 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:42.375+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:42.451 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:42.448+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:42.535 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:42.530+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:42.720 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:42.713+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:42.721 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:42.716+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:42.751 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:42.745+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:42.753 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:42.750+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:42.956 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:42.951+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:43.272 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:43.263+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:43.412 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:43.406+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:43.473 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:43.470+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:43.506 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:43.500+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:43.689 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:43.683+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:43.690 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:43.685+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:43.766 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:43.762+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:43.768 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:43.765+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:43.921 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:43.915+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:44.240 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:44.237+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:44.444 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:44.438+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:44.471 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:44.468+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:44.544 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:44.538+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:44.676 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:44.669+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:44.676 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:44.672+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:44.724 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:44.722+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:44.729 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:44.727+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:44.936 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:44.930+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:45.243 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:45.240+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:45.402 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:45.396+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:45.485 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:45.475+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:45.552 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:45.543+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:45.712 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:45.706+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:45.713 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:45.709+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:45.714 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:45.710+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:45.716 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:45.713+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:45.942 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:45.936+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:46.266 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:46.261+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:46.439 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:46.432+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:46.478 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:46.474+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:46.601 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:46.583+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:46.748 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:46.742+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:46.749 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:46.744+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:46.760 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:46.756+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:46.762 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:46.759+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:46.990 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:46.984+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:47.223 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:47.219+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:47.426 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:47.420+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:47.435 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:47.433+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:47.604 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:47.599+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:47.721 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:47.715+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:47.723 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:47.717+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:47.770 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:47.769+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:47.773 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:47.771+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:47.894 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:29:47.888+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e301 v301) 2019-12-10T02:29:47.975 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:47.968+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:48.226 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:48.224+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:48.402 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:48.396+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:48.476 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:48.473+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:48.639 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:48.633+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:48.731 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:48.726+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:48.733 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:48.730+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:48.808 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:48.805+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:48.809 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:48.807+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:49.006 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:49.000+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:49.231 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:49.229+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:49.399 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:49.394+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:49.481 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:49.478+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:49.692 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:49.686+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:49.743 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:49.736+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:49.743 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:49.739+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:49.791 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:49.788+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:49.796 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:49.795+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:49.970 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:49.964+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:50.217 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:50.211+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:50.387 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:50.381+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:50.458 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:50.456+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:50.698 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:50.692+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:50.746 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:50.739+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:50.746 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:50.742+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:50.819 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:50.817+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:50.823 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:50.821+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:50.947 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:50.940+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:51.222 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:51.218+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:51.397 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:51.391+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:51.413 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:51.411+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:51.688 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:51.682+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:51.758 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:51.752+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:51.759 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:51.755+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:51.875 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:51.869+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:51.876 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:51.873+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:51.967 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:51.958+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:52.205 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:52.202+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:52.422 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:52.415+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:52.440 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:52.437+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:52.739 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:52.732+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:52.811 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:52.806+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:52.813 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:52.809+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:52.904 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:29:52.903+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 5 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e301 v301) 2019-12-10T02:29:52.921 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:52.916+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:52.921 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:52.919+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:52.980 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:52.974+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:53.229 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:53.224+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:53.402 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:53.399+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:53.453 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:53.447+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:53.742 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:53.736+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:53.850 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:53.843+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:53.851 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:53.846+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:53.883 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:53.878+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:53.884 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:53.882+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:53.984 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:53.978+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:54.234 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:54.231+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:54.402 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:54.399+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:54.490 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:54.485+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:54.740 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:54.733+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:54.818 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:54.812+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:54.818 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:54.814+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:54.892 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:54.890+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:54.895 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:54.894+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:54.948 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:54.942+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:55.229 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:55.225+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:55.372 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:55.368+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:55.462 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:55.456+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:55.727 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:55.721+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:55.797 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:55.789+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:55.797 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:55.792+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:55.909 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:55.905+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:55.914 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:55.911+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:55.978 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:55.973+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:56.202 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:56.197+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:56.353 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:56.349+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:56.427 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:56.421+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:56.742 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:56.737+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:56.823 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:56.818+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:56.826 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:56.823+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:56.911 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:56.906+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:56.913 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:56.908+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:57.026 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:57.020+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:57.204 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:57.201+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:57.317 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:57.314+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:57.408 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:57.400+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:57.722 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:57.715+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:57.822 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:57.816+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:57.887 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:57.818+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:57.922 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:29:57.919+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e301 v301) 2019-12-10T02:29:57.932 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:57.928+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:57.934 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:57.930+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:57.997 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:57.987+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:58.176 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:58.172+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:58.302 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:58.299+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:58.386 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:58.380+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:58.700 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:58.695+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:58.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:58.801+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:58.808 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:58.804+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:58.958 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:58.952+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:58.982 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:58.978+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:58.984 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:58.981+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:29:59.160 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:29:59.156+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:29:59.318 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:29:59.314+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:29:59.415 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:29:59.406+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:29:59.743 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:29:59.736+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:29:59.798 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:59.791+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:29:59.798 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:29:59.794+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:29:59.924 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:29:59.919+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:29:59.995 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:59.991+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:29:59.997 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:29:59.993+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:00.167 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:00.163+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:00.313 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:00.309+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:00.443 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:00.437+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:00.754 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:00.747+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:00.791 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:00.786+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:00.797 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:00.791+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:00.957 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:00.951+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:00.960 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:00.956+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:00.966 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:00.962+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:01.151 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:01.146+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:01.325 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:01.321+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:01.466 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:01.461+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:01.776 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:01.769+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:01.836 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:01.830+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:01.839 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:01.835+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:01.936 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:01.932+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:01.939 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:01.936+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:01.949 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:01.944+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:02.157 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:02.154+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:02.305 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:02.301+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:02.489 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:02.484+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:02.759 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:02.752+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:02.801 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:02.795+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:02.804 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:02.800+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:02.929 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:02.926+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:02.933 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:02.930+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:02.935 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:30:02.931+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e301 v301) 2019-12-10T02:30:02.957 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:02.951+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:03.114 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:03.110+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:03.354 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:03.350+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:03.539 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:03.526+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:03.770 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:03.763+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:03.837 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:03.832+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:03.839 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:03.835+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:03.891 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:03.886+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:03.892 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:03.889+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:03.965 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:03.959+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:04.119 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:04.115+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:04.346 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:04.343+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:04.582 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:04.575+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:04.775 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:04.771+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:04.847 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:04.843+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:04.850 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:04.847+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:04.853 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:04.850+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:04.855 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:04.852+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:04.935 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:04.930+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:05.169 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:05.166+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:05.384 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:05.378+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:05.608 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:05.602+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:05.813 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:05.807+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:05.850 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:05.844+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:05.851 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:05.847+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:05.880 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:05.878+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:05.885 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:05.881+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:05.944 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:05.937+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:06.187 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:06.183+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:06.398 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:06.393+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:06.623 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:06.616+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:06.827 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:06.821+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:06.844 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:06.838+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:06.845 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:06.840+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:06.898 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:06.893+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:06.900 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:06.896+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:06.971 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:06.967+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:07.144 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:07.141+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:07.432 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:07.427+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:07.598 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:07.592+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:07.839 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:07.833+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:07.841 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:07.838+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:07.863 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:07.857+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:07.921 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:07.916+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:07.922 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:07.919+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:07.950 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:30:07.947+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e301 v301) 2019-12-10T02:30:07.969 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:07.964+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:08.171 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:08.167+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:08.417 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:08.412+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:08.633 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:08.627+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:08.860 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:08.855+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:08.863 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:08.859+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:08.902 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:08.899+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:08.912 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:08.907+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:08.915 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:08.909+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:08.993 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:08.987+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:09.203 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:09.200+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:09.463 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:09.458+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:09.649 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:09.644+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:09.857 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:09.852+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:09.860 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:09.856+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:09.880 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:09.875+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:09.885 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:09.882+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:09.921 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:09.916+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:10.037 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:10.031+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:10.198 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:10.195+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:10.483 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:10.480+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:10.632 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:10.628+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:10.889 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:10.885+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:10.891 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:10.885+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:10.892 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:10.887+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:10.892 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:10.887+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:10.895 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:10.889+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:10.993 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:10.987+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:11.205 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:11.200+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:11.444 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:11.440+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:11.653 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:11.646+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:11.916 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:11.905+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:11.918 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:11.911+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:11.923 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:11.917+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:11.928 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:11.923+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:11.929 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:11.925+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:12.010 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:12.005+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:12.229 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:30:12.224+0000 1694f700 -1 osd.0 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.308492+0000 front 2019-12-10T02:28:34.271196+0000 (oldest deadline 2019-12-10T02:29:19.567558+0000) 2019-12-10T02:30:12.475 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:30:12.472+0000 1694f700 -1 osd.2 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.554178+0000 front 2019-12-10T02:28:34.557491+0000 (oldest deadline 2019-12-10T02:29:16.244469+0000) 2019-12-10T02:30:12.677 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:30:12.672+0000 1694f700 -1 osd.7 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.342714+0000 front 2019-12-10T02:28:33.226908+0000 (oldest deadline 2019-12-10T02:29:19.071676+0000) 2019-12-10T02:30:12.892 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:12.885+0000 1694f700 -1 osd.5 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:33.079055+0000 front 2019-12-10T02:28:32.988246+0000 (oldest deadline 2019-12-10T02:29:16.448123+0000) 2019-12-10T02:30:12.893 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:30:12.888+0000 1694f700 -1 osd.5 301 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:109622 2.2s0 2:433373a1:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_6af6d28d3ab8468ba6adf2b5c2de4fd6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e296) 2019-12-10T02:30:12.909 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:12.903+0000 1694f700 -1 osd.3 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 ever on either front or back, first ping sent 2019-12-10T02:28:55.234514+0000 (oldest deadline 2019-12-10T02:29:35.234514+0000) 2019-12-10T02:30:12.910 INFO:tasks.ceph.osd.3.smithi067.stderr:2019-12-10T02:30:12.907+0000 1694f700 -1 osd.3 301 get_health_metrics reporting 18 slow ops, oldest is osd_op(client.4563.0:87135 7.2 7.45971ef2 (undecoded) ondisk+retry+read+known_if_redirected e300) 2019-12-10T02:30:12.959 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:30:12.953+0000 1694f700 -1 osd.6 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:30.869715+0000 front 2019-12-10T02:28:30.872118+0000 (oldest deadline 2019-12-10T02:29:15.456968+0000) 2019-12-10T02:30:13.018 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:30:13.011+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 44sec e301 v301) 2019-12-10T02:30:13.044 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:30:13.037+0000 1694f700 -1 osd.4 301 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:28:34.106797+0000 front 2019-12-10T02:28:34.124745+0000 (oldest deadline 2019-12-10T02:29:18.191915+0000) 2019-12-10T02:30:42.135 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:30:42.131+0000 114ae700 -1 osd.1 306 build_incremental_map_msg missing incremental map 306 2019-12-10T02:30:42.177 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:30:42.149+0000 36f91700 -1 osd.1 306 build_incremental_map_msg missing incremental map 306 2019-12-10T02:30:42.407 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:30:42.270+0000 37f93700 -1 osd.1 306 build_incremental_map_msg missing incremental map 306 2019-12-10T02:30:42.419 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:30:42.273+0000 37f93700 -1 osd.1 306 build_incremental_map_msg missing incremental map 306 2019-12-10T02:30:42.426 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:30:42.314+0000 37f93700 -1 osd.1 306 build_incremental_map_msg missing incremental map 306 2019-12-10T02:31:05.122 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:05.118+0000 35f8f700 -1 osd.1 307 build_incremental_map_msg missing incremental map 306 2019-12-10T02:31:11.610 INFO:teuthology.orchestra.run.smithi067.stderr:testIfMatchAndUnmodified (test.functional.tests.TestFileComparison) ... ok 2019-12-10T02:31:42.362 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:42.358+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:42.983 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:42.978+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:43.278 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:43.275+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:43.289 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:43.283+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:43.341 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:43.336+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:43.999 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:43.993+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:44.249 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:44.243+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:44.315 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:44.311+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:44.316 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:44.313+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:44.770 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:44.767+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:45.005 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:45.000+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:45.287 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:45.284+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:45.296 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:45.290+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:45.329 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:45.325+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:45.821 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:45.818+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:45.965 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:45.960+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:46.272 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:46.268+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:46.313 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:46.309+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:46.322 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:46.316+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:46.323 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:46.319+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:46.824 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:46.820+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:47.001 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:46.995+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:47.023 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:47.015+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:47.293 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:47.289+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:47.352 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:47.349+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:47.369 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:47.363+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:47.369 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:47.365+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:47.574 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:47.569+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:47.790 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:47.785+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:48.054 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:48.047+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:48.067 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:48.061+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:48.343 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:48.338+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:48.344 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:48.340+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:48.355 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:48.349+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:48.356 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:48.351+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:48.606 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:48.600+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:48.771 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:48.767+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:49.018 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:49.013+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:49.128 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:49.123+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:49.311 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:49.308+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:49.354 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:49.351+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:49.365 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:49.359+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:49.365 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:49.361+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:49.620 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:49.615+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:49.806 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:49.802+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:50.034 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:50.026+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:50.129 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:50.120+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:50.293 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:50.289+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:50.384 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:50.380+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:50.389 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:50.384+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:50.393 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:50.388+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:50.642 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:50.636+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:50.859 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:50.856+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:50.999 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:50.993+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:51.116 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:51.110+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:51.289 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:51.285+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:51.352 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:51.345+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:51.352 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:51.347+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:51.361 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:51.352+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:51.603 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:51.598+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:51.893 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:51.889+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:52.033 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:52.028+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:52.170 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:52.155+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:52.331 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:52.325+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:52.332 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:52.328+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:52.332 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:52.327+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:52.351 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:52.348+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:52.625 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:52.619+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:52.860 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:52.857+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:53.060 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:53.054+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:53.134 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:53.128+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:53.311 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:53.306+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:53.314 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:53.310+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:53.340 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:53.335+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:53.342 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:53.338+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:53.647 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:53.642+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:53.861 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:53.857+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:54.071 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:54.066+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:54.153 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:54.147+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:54.327 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:54.321+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:54.328 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:54.323+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:54.340 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:54.336+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:54.365 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:54.362+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:54.617 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:54.608+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:54.834 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:54.831+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:55.122 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:55.116+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:55.161 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:55.155+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:55.310 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:55.304+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:55.311 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:55.306+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:55.355 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:55.352+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:55.406 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:55.404+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:55.643 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:55.638+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:55.834 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:55.832+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:56.132 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:56.126+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:56.139 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:56.134+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:56.271 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:56.265+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:56.271 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:56.267+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:56.379 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:56.377+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:56.409 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:56.405+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:56.634 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:56.623+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:56.844 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:56.842+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:57.089 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:57.083+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:57.099 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:57.094+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:57.265 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:57.260+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:57.270 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:57.266+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:57.354 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:57.351+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:57.368 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:57.365+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:57.631 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:57.626+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:57.868 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:57.866+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:58.058 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:58.052+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:58.107 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:58.100+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:58.272 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:58.267+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:58.276 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:58.272+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:58.329 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:58.327+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:58.419 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:58.417+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:58.592 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:58.587+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:58.831 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:58.828+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:31:59.095 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:31:59.090+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:31:59.102 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:31:59.096+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:31:59.243 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:59.237+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:31:59.244 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:31:59.239+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:31:59.303 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:31:59.299+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:31:59.452 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:31:59.449+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:31:59.605 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:31:59.600+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:31:59.874 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:31:59.871+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:00.056 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:00.051+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:00.069 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:00.063+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:00.201 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:00.195+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:00.202 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:00.197+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:00.302 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:00.299+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:00.495 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:00.492+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:00.569 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:00.563+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:00.915 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:00.913+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:01.049 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:01.043+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:01.093 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:01.087+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:01.255 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:01.250+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:01.261 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:01.254+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:01.353 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:01.347+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:01.515 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:01.512+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:01.596 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:01.590+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:01.914 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:01.911+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:02.053 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:02.048+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:02.060 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:02.055+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:02.247 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:02.242+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:02.251 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:02.247+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:02.373 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:02.370+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:02.522 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:02.520+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:02.621 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:02.616+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:02.937 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:02.933+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:03.082 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:03.076+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:03.095 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:03.089+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:03.265 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:03.258+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:03.265 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:03.260+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:03.370 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:03.368+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:03.544 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:03.541+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:03.658 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:03.652+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:03.921 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:03.917+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:04.067 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:04.061+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:04.080 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:04.073+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:04.223 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:04.217+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:04.227 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:04.222+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:04.381 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:04.379+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:04.543 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:04.540+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:04.673 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:04.667+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:04.937 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:04.934+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:05.085 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:05.079+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:05.115 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:05.110+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:05.221 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:05.214+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:05.221 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:05.217+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:05.429 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:05.427+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:05.529 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:05.526+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:05.663 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:05.657+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:05.929 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:05.926+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:06.089 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:06.084+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:06.166 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:06.157+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:06.197 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:06.192+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:06.199 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:06.194+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:06.455 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:06.453+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:06.570 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:06.568+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:06.633 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:06.627+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:06.928 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:06.925+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:07.049 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:07.043+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:07.170 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:07.164+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:07.182 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:07.177+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:07.185 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:07.181+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:07.457 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:07.454+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:07.595 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:07.589+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:07.597 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:07.595+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:07.931 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:07.929+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:08.069 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:08.064+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:08.160 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:08.155+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:08.208 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:08.203+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:08.209 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:08.205+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:08.452 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:08.449+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:08.566 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:08.560+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:08.631 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:08.628+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:08.933 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:08.930+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:09.109 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:09.103+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:09.141 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:09.135+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:09.228 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:09.223+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:09.231 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:09.227+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:09.477 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:09.473+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:09.606 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:09.600+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:09.662 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:09.658+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:09.895 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:09.891+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:10.092 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:10.086+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:10.186 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:10.180+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:10.216 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:10.211+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:10.219 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:10.215+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:10.466 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:10.461+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:10.616 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:10.613+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:10.624 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:10.619+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:10.893 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:10.890+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:11.129 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:11.125+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:11.224 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:11.219+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:11.266 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:11.261+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:11.269 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:11.265+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:11.468 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:11.465+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:11.617 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:11.614+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:11.653 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:11.648+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:11.922 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:11.919+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:12.115 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:12.105+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:12.194 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:12.189+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:12.242 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:12.236+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:12.243 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:12.239+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:12.524 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:12.517+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:12.654 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:12.651+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:12.699 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:12.693+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:12.880 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:12.875+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:13.156 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:13.150+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:13.238 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:13.232+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:13.238 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:13.234+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:13.241 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:13.236+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:13.525 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:13.522+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:13.679 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:13.672+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:13.743 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:13.734+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:13.883 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:13.874+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:14.137 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:14.131+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:14.240 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:14.235+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:14.261 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:14.254+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:14.262 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:14.257+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:14.548 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:14.544+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:14.662 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:14.659+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:14.751 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:14.744+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:14.792 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:58:35.855 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:32:14.801 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:58:35.864 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:32:14.815 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:58:35.878 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:32:14.818 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:58:35.881 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:32:14.829 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:58:35.988 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:32:14.840 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:14.837+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:14.855 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:58:36.014 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:32:15.098 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:05:58:36.161 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:32:15.152 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:15.144+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:15.177 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:05:58:36.238 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:32:15.203 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:05:58:36.362 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:32:15.258 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:15.252+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:15.258 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:15.254+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:15.267 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:15.261+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:15.549 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:15.546+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:15.679 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:15.677+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:15.778 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:15.768+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:15.866 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:15.862+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:16.193 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:16.187+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:16.220 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:16.215+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:16.222 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:16.219+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:16.279 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:16.274+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:16.517 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:16.514+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:16.642 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:16.638+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:16.731 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:16.725+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:16.836 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:16.834+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:17.212 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:17.200+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:17.213 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:17.204+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:17.251 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:17.241+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:17.332 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:17.326+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:17.547 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:17.544+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:17.683 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:17.680+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:17.710 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:17.706+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:17.881 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:17.878+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:18.240 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:18.234+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:18.241 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:18.236+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:18.260 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:18.254+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:18.360 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:18.354+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:18.366 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:32:18.360+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 46sec e307 v307) 2019-12-10T02:32:18.579 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:18.574+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:18.722 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:18.718+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:18.736 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:18.729+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:18.858 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:18.854+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:19.235 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:19.230+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:19.238 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:19.234+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:19.312 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:19.307+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:19.342 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:19.337+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:19.553 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:19.550+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:19.737 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:19.733+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:19.789 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:19.783+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:19.903 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:19.900+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:20.198 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:20.192+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:20.199 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:20.194+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:20.288 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:20.282+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:20.324 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:20.318+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:20.552 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:20.549+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:20.746 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:20.741+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:20.769 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:20.765+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:20.880 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:20.877+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:21.215 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:21.209+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:21.218 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:21.214+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:21.284 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:21.279+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:21.359 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:21.354+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:21.585 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:21.581+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:21.735 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:21.732+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:21.754 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:21.749+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:21.922 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:21.919+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:22.204 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:22.199+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:22.205 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:22.201+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:22.316 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:22.311+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:22.378 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:22.374+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:22.542 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:22.540+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:22.735 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:32:22.731+0000 1694f700 -1 osd.2 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.464636+0000 front 2019-12-10T02:30:57.367699+0000 (oldest deadline 2019-12-10T02:31:42.357515+0000) 2019-12-10T02:32:22.757 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:32:22.751+0000 1694f700 -1 osd.6 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.002474+0000 front 2019-12-10T02:31:00.834428+0000 (oldest deadline 2019-12-10T02:31:46.678189+0000) 2019-12-10T02:32:22.970 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:32:22.966+0000 1694f700 -1 osd.0 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.283857+0000 front 2019-12-10T02:31:01.206855+0000 (oldest deadline 2019-12-10T02:31:44.020848+0000) 2019-12-10T02:32:23.234 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:23.229+0000 1694f700 -1 osd.5 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:57.049528+0000 front 2019-12-10T02:30:56.821736+0000 (oldest deadline 2019-12-10T02:31:42.493637+0000) 2019-12-10T02:32:23.237 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:23.233+0000 1694f700 -1 osd.5 307 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:23.304 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:32:23.298+0000 1694f700 -1 osd.4 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:31:01.834746+0000 front 2019-12-10T02:31:01.960878+0000 (oldest deadline 2019-12-10T02:31:46.960101+0000) 2019-12-10T02:32:23.384 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:32:23.378+0000 1694f700 -1 osd.7 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:58.503296+0000 front 2019-12-10T02:30:58.609731+0000 (oldest deadline 2019-12-10T02:31:42.574438+0000) 2019-12-10T02:32:23.431 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:32:23.428+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 7 slow ops, oldest is osd_failure(failed timeout osd.3 [v2:172.21.15.67:6818/10704,v1:172.21.15.67:6819/10704] for 46sec e307 v307) 2019-12-10T02:32:23.530 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:32:23.527+0000 1694f700 -1 osd.1 307 heartbeat_check: no reply from 172.21.15.67:6822 osd.3 since back 2019-12-10T02:30:59.527890+0000 front 2019-12-10T02:30:59.577812+0000 (oldest deadline 2019-12-10T02:31:42.281765+0000) 2019-12-10T02:32:24.639 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:24.636+0000 1694f700 -1 osd.5 308 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:25.777 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:25.771+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:26.748 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:26.744+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:27.720 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:27.717+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:28.680 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:28.676+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:29.655 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:29.650+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:30.680 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:30.676+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:31.652 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:31.648+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:32.660 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:32.656+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:33.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:33.680+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:34.677 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:34.674+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:35.663 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:35.659+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:36.667 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:36.663+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:37.665 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:37.661+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:38.646 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:38.642+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:39.614 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:39.609+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:40.638 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:40.634+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:41.642 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:41.638+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:42.693 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:42.686+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:43.649 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:43.645+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:44.650 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:44.646+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:45.653 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:45.648+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:46.617 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:46.612+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:47.588 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:47.584+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:48.616 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:48.612+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:49.625 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:49.620+0000 1694f700 -1 osd.5 309 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:50.603 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:50.600+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:51.580 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:51.576+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:52.575 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:52.570+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:53.581 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:53.577+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:54.567 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:54.560+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:55.610 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:55.605+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:56.591 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:56.586+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:57.639 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:57.635+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:58.600 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:58.595+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:32:59.571 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:32:59.566+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:00.536 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:00.533+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:01.521 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:01.517+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:02.530 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:02.526+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:03.550 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:03.545+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:04.557 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:04.553+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:05.588 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:05.584+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:06.579 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:06.574+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:07.562 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:07.558+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:08.530 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:08.526+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:09.534 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:09.530+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:10.504 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:10.501+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:11.468 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:11.464+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:12.432 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:12.428+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:13.420 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:13.412+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:14.417 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:14.412+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:15.468 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:15.464+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:16.455 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:16.450+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:17.506 INFO:tasks.ceph.osd.3.smithi067.stderr:*** Caught signal (Aborted) ** 2019-12-10T02:33:17.506 INFO:tasks.ceph.osd.3.smithi067.stderr: in thread 31f87700 thread_name:tp_osd_tp 2019-12-10T02:33:17.507 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:17.497+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:17.566 INFO:tasks.ceph.osd.3.smithi067.stderr: ceph version 15.0.0-7936-g827e03c (827e03c8a6b1e17d93dcaf79724437bac291c35b) octopus (dev) 2019-12-10T02:33:17.567 INFO:tasks.ceph.osd.3.smithi067.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-10T02:33:17.567 INFO:tasks.ceph.osd.3.smithi067.stderr: 2: (pthread_kill()+0x31) [0xc09ba61] 2019-12-10T02:33:17.567 INFO:tasks.ceph.osd.3.smithi067.stderr: 3: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, unsigned long)+0x244) [0xcb33a4] 2019-12-10T02:33:17.567 INFO:tasks.ceph.osd.3.smithi067.stderr: 4: (ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, unsigned long, unsigned long)+0x252) [0xcb3972] 2019-12-10T02:33:17.567 INFO:tasks.ceph.osd.3.smithi067.stderr: 5: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x446) [0xcd2276] 2019-12-10T02:33:17.567 INFO:tasks.ceph.osd.3.smithi067.stderr: 6: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xcda690] 2019-12-10T02:33:17.567 INFO:tasks.ceph.osd.3.smithi067.stderr: 7: (()+0x7e65) [0xc096e65] 2019-12-10T02:33:17.568 INFO:tasks.ceph.osd.3.smithi067.stderr: 8: (clone()+0x6d) [0xd20e88d] 2019-12-10T02:33:18.462 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:18.458+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:19.513 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:19.508+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:20.486 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:20.481+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:21.471 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:21.466+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:21.472 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:21.468+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:22.192 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:22.187+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:22.259 INFO:tasks.ceph.osd.3.smithi067.stderr:daemon-helper: command crashed with signal 6 2019-12-10T02:33:22.447 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:22.443+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:22.454 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:22.448+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:22.455 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:22.450+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:22.699 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:22.693+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:23.198 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:23.192+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:23.418 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:23.415+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:23.502 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:23.496+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:23.506 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:23.502+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:23.729 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:23.718+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:24.236 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:24.231+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:24.373 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:24.369+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:24.537 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:24.530+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:24.537 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:24.533+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:24.718 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:24.713+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:24.844 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:24.842+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:24.884 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:24.879+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:25.195 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:25.188+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:25.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2019-12-10T02:33:25.406 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:25.403+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:25.558 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:25.553+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:25.561 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:25.557+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:25.739 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:25.733+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:25.800 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:25.796+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:25.842 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:25.835+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:26.237 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:26.232+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:26.359 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:26.356+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:26.586 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:26.580+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:26.588 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:26.584+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:26.722 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:26.716+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:26.812 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:26.809+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:26.839 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:26.832+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:27.245 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:27.240+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:27.364 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:27.361+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:27.564 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:27.558+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:27.564 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:27.560+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:27.748 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:27.741+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:27.824 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:27.818+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:27.854 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:27.851+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:28.227 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:28.222+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:28.328 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:28.324+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:28.608 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:28.602+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:28.609 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:28.604+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:28.746 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:28.740+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:28.852 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:28.846+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:28.868 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:28.865+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:29.195 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:29.189+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:29.320 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:29.317+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:29.587 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:29.582+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:29.591 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:29.587+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:29.781 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:29.773+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:29.858 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:29.856+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:29.880 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:29.874+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:30.208 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:30.203+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:30.302 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:30.299+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:30.643 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:30.635+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:30.644 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:30.638+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:30.814 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:30.811+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:30.832 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:30.827+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:30.927 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:30.922+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:31.262 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:31.257+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:31.307 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:31.304+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:31.509 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~6s 2019-12-10T02:33:31.663 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:31.658+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:31.666 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:31.662+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:31.823 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:31.816+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:31.833 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:31.829+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:31.908 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:31.902+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:32.222 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:32.217+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:32.312 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:32.308+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:32.713 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:32.707+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:32.714 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:32.710+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:32.813 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:32.808+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:32.870 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:32.868+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:32.912 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:32.907+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:33.224 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:33.219+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:33.335 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:33.333+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:33.720 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:33.714+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:33.720 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:33.716+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:33.829 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:33.823+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:33.855 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:33.853+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:33.903 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:33.898+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:34.224 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:34.219+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:34.377 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:34.375+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:34.722 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:34.716+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:34.722 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:34.718+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:34.873 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:34.867+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:34.896 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:34.894+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:34.938 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:34.932+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:35.259 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:35.254+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:35.345 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:35.343+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:35.687 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:35.682+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:35.690 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:35.686+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:35.854 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:35.851+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:35.865 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:35.859+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:35.942 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:35.936+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:36.223 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:36.217+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:36.313 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:36.310+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:36.699 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:36.693+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:36.699 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:36.695+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:36.870 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:36.868+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:36.891 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:36.886+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:36.917 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:36.911+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:37.210 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:37.204+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:37.328 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:37.326+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:37.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:37.679+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:37.687 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:37.683+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:37.820 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~13s 2019-12-10T02:33:37.912 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:37.907+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:37.921 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:37.917+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:37.933 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:37.927+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:38.179 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:38.173+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:38.304 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:38.301+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:38.706 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:38.700+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:38.707 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:38.703+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:38.902 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:38.896+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:38.931 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:38.925+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:38.936 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:38.934+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:39.189 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:39.183+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:39.282 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:39.279+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:39.736 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:39.730+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:39.737 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:39.733+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:39.903 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:39.898+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:39.936 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:39.930+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:39.977 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:39.972+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:40.219 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:40.214+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:40.328 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:40.324+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:40.732 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:40.726+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:40.732 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:40.728+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:40.888 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:40.886+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:40.953 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:40.947+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:41.034 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:41.022+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:41.191 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:41.185+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:41.313 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:41.310+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:41.758 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:41.751+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:41.758 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:41.754+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:41.843 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:41.840+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:41.991 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:41.985+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:42.057 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:42.052+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:42.168 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:42.163+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:42.289 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:42.285+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:42.722 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:42.717+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:42.723 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:42.719+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:42.845 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:42.841+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:43.039 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:43.032+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:43.097 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:43.091+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:43.222 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:43.216+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:43.316 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:43.311+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:43.691 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:43.685+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:43.693 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:43.687+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:43.865 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:43.862+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:44.076 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:44.070+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:44.086 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:44.081+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:44.128 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~19s 2019-12-10T02:33:44.181 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:44.175+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:44.312 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:44.307+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:44.659 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:44.653+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:44.659 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:44.655+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:44.917 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:44.910+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:45.073 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:45.067+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:45.139 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:45.135+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:45.184 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:45.177+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:45.341 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:45.337+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:45.635 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:45.629+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:45.641 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:45.634+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:45.933 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:45.929+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:46.047 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:46.040+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:46.132 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:46.126+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:46.236 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:46.229+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:46.379 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:46.376+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:46.656 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:46.649+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:46.657 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:46.652+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:46.914 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:46.911+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:47.079 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:47.072+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:47.176 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:47.168+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:47.258 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:47.252+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:47.415 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:47.410+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:47.633 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:47.627+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:47.634 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:47.629+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:47.940 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:47.935+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:48.123 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:33:48.118+0000 1694f700 -1 osd.7 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.059253+0000 front 2019-12-10T02:32:37.042823+0000 (oldest deadline 2019-12-10T02:33:22.272400+0000) 2019-12-10T02:33:48.182 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:33:48.176+0000 1694f700 -1 osd.6 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:40.618207+0000 front 2019-12-10T02:32:40.659714+0000 (oldest deadline 2019-12-10T02:33:24.691086+0000) 2019-12-10T02:33:48.256 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:33:48.251+0000 1694f700 -1 osd.4 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:36.367714+0000 front 2019-12-10T02:32:36.319290+0000 (oldest deadline 2019-12-10T02:33:22.175627+0000) 2019-12-10T02:33:48.426 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:33:48.423+0000 1694f700 -1 osd.2 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:37.062819+0000 front 2019-12-10T02:32:37.018158+0000 (oldest deadline 2019-12-10T02:33:22.288826+0000) 2019-12-10T02:33:48.639 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:48.633+0000 1694f700 -1 osd.5 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:41.126262+0000 front 2019-12-10T02:32:36.435223+0000 (oldest deadline 2019-12-10T02:33:21.044607+0000) 2019-12-10T02:33:48.640 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:33:48.636+0000 1694f700 -1 osd.5 310 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs,stat,read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e307) 2019-12-10T02:33:48.923 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:33:48.920+0000 1694f700 -1 osd.0 310 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:32:39.175989+0000 front 2019-12-10T02:32:39.268408+0000 (oldest deadline 2019-12-10T02:33:24.665181+0000) 2019-12-10T02:33:50.447 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~25s 2019-12-10T02:33:56.759 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~32s 2019-12-10T02:34:03.069 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~38s 2019-12-10T02:34:04.245 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:00:25.308 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:04.249 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:00:25.313 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:04.304 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:00:25.365 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:04.312 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:00:25.375 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:04.339 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:00:25.498 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:04.343 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:00:25.502 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:04.402 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:00:25.465 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:04.429 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:00:25.492 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:04.517 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:00:25.675 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:09.380 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~44s 2019-12-10T02:34:15.690 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~50s 2019-12-10T02:34:22.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~57s 2019-12-10T02:34:28.309 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~63s 2019-12-10T02:34:29.469 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:00:50.532 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:29.521 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:00:50.584 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:29.547 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:00:50.610 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:29.552 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:00:50.615 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:29.580 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:00:50.643 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:29.583 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:00:50.743 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:29.589 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:00:50.748 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:29.594 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:00:50.657 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:29.631 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:00:50.791 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:34:34.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~69s 2019-12-10T02:34:40.930 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~76s 2019-12-10T02:34:41.556 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:41.458+0000 36790700 -1 osd.1 315 build_incremental_map_msg missing incremental map 315 2019-12-10T02:34:41.650 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:41.474+0000 36790700 -1 osd.1 315 build_incremental_map_msg missing incremental map 315 2019-12-10T02:34:41.681 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:41.487+0000 36790700 -1 osd.1 315 build_incremental_map_msg missing incremental map 315 2019-12-10T02:34:41.727 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:41.581+0000 32f89700 -1 osd.1 315 build_incremental_map_msg missing incremental map 315 2019-12-10T02:34:41.832 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:41.707+0000 3578e700 -1 osd.1 315 build_incremental_map_msg missing incremental map 315 2019-12-10T02:34:42.353 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:42.351+0000 3578e700 -1 osd.1 315 build_incremental_map_msg missing incremental map 315 2019-12-10T02:34:42.501 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:42.462+0000 3578e700 -1 osd.1 316 build_incremental_map_msg missing incremental map 316 2019-12-10T02:34:42.706 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:42.520+0000 31786700 -1 osd.1 316 build_incremental_map_msg missing incremental map 316 2019-12-10T02:34:42.958 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:42.732+0000 38794700 -1 osd.1 316 build_incremental_map_msg missing incremental map 316 2019-12-10T02:34:43.052 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:42.903+0000 37f93700 -1 osd.1 316 build_incremental_map_msg missing incremental map 316 2019-12-10T02:34:43.056 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:42.930+0000 32788700 -1 osd.1 316 build_incremental_map_msg missing incremental map 316 2019-12-10T02:34:45.373 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:34:45.334+0000 32788700 -1 osd.1 316 build_incremental_map_msg missing incremental map 316 2019-12-10T02:34:45.382 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-10T02:34:45.347+0000 20e25700 -1 RGWWatcher::handle_error cookie 5342034000 err (107) Transport endpoint is not connected 2019-12-10T02:34:47.241 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~82s 2019-12-10T02:34:53.551 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~88s 2019-12-10T02:34:59.864 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~95s 2019-12-10T02:35:06.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~101s 2019-12-10T02:35:09.795 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:09.791+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:10.774 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:10.770+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:11.732 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:11.728+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:12.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~107s 2019-12-10T02:35:12.783 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:12.779+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:13.758 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:13.754+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:14.724 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:14.721+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:15.695 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:15.691+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:16.677 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:16.672+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:17.696 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:17.692+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:18.707 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:18.703+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:18.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~114s 2019-12-10T02:35:19.683 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:19.679+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:20.664 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:20.660+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:21.657 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:21.653+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:22.644 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:22.640+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:23.615 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:23.611+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:24.660 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:24.656+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:25.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~120s 2019-12-10T02:35:25.618 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:25.613+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:26.635 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:26.631+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:27.680 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:27.676+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:28.663 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:28.659+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:29.693 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:29.688+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:30.703 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:30.700+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:31.414 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~126s 2019-12-10T02:35:31.674 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:31.670+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:32.684 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:32.680+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:33.717 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:33.711+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:34.742 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:34.737+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:35.717 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:35.713+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:36.702 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:36.698+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:37.709 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:37.705+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:37.723 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~133s 2019-12-10T02:35:38.758 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:38.754+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:39.726 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:39.722+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:40.763 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:40.758+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:41.789 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:41.786+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:42.824 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:42.820+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:43.866 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:43.862+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:44.039 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~139s 2019-12-10T02:35:44.921 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:44.913+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:45.881 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:45.875+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:46.903 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:46.899+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:47.936 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:47.932+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:48.912 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:48.907+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:49.881 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:49.877+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:50.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~145s 2019-12-10T02:35:50.896 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:50.892+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:51.943 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:51.939+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:52.957 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:52.948+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:53.981 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:53.977+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:55.027 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:55.023+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:56.009 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:56.006+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:56.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~151s 2019-12-10T02:35:56.966 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:56.961+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:57.931 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:57.928+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:58.962 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:58.958+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:35:59.953 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:35:59.950+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:00.938 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:00.933+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:01.918 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:01.914+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:02.970 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:02.966+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:02.980 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~158s 2019-12-10T02:36:03.948 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:03.944+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:04.995 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:04.991+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:05.989 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:05.982+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:07.004 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:07.001+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:07.990 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:07.985+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:08.958 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:02:30.021 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:36:08.964 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:02:30.027 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:36:08.975 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:02:30.038 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:36:08.978 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:02:30.042 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:36:08.985 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:02:30.144 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:36:08.991 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:02:30.150 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:36:08.993 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:08.988+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:09.100 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:02:30.163 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:36:09.118 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:02:30.181 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:36:09.157 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:02:30.316 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:36:09.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~164s 2019-12-10T02:36:10.031 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:10.027+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:11.078 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:11.073+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:12.068 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:12.060+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:13.046 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:13.042+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:14.001 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:13.997+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:15.010 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:15.006+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:15.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~170s 2019-12-10T02:36:16.022 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:16.018+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:16.993 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:16.989+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:18.044 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:18.040+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:19.036 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:19.032+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:20.068 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:20.064+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:21.108 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:21.103+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:21.916 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~177s 2019-12-10T02:36:22.071 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:22.067+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:23.053 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:23.049+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:24.091 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:24.088+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:25.111 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:25.107+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:26.087 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:26.083+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:27.129 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:27.125+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:27.484 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:27.481+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:28.163 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:28.160+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:28.224 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~183s 2019-12-10T02:36:28.476 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:28.473+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:28.690 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:28.684+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:29.198 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:29.192+0000 1694f700 -1 osd.5 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.880881+0000 front 2019-12-10T02:35:43.817626+0000 (oldest deadline 2019-12-10T02:36:28.178227+0000) 2019-12-10T02:36:29.199 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:29.195+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:29.423 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:29.417+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:29.495 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:29.491+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:29.696 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:29.690+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:29.855 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:29.852+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:30.184 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:30.179+0000 1694f700 -1 osd.5 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.880881+0000 front 2019-12-10T02:35:43.817626+0000 (oldest deadline 2019-12-10T02:36:28.178227+0000) 2019-12-10T02:36:30.187 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:30.183+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:30.450 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:30.448+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:30.477 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:30.471+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:30.661 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:30.651+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:30.820 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:30.816+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:31.092 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:31.088+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:31.175 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:31.170+0000 1694f700 -1 osd.5 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.880881+0000 front 2019-12-10T02:35:43.817626+0000 (oldest deadline 2019-12-10T02:36:28.178227+0000) 2019-12-10T02:36:31.178 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:31.174+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:31.435 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:31.432+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:31.506 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:31.500+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:31.634 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:31.625+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:31.789 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:31.784+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:32.112 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:32.107+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:32.188 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:32.182+0000 1694f700 -1 osd.5 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.880881+0000 front 2019-12-10T02:35:43.817626+0000 (oldest deadline 2019-12-10T02:36:28.178227+0000) 2019-12-10T02:36:32.189 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:32.185+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:32.432 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:32.428+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:32.489 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:32.483+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:32.684 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:32.678+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:32.825 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:32.820+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:33.091 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:33.086+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:33.145 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:33.141+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:33.416 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:33.413+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:33.537 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:33.530+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:33.657 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:33.652+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:33.795 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:33.792+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:34.055 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:34.050+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:34.161 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:34.156+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:34.453 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:34.450+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:34.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~189s 2019-12-10T02:36:34.585 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:34.579+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:34.699 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:34.693+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:34.776 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:34.773+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:35.052 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:35.046+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:35.186 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:35.181+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:35.448 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:35.445+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:35.633 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:35.627+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:35.742 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:35.740+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:35.753 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:35.749+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:36.016 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:36.010+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:36.189 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:36.185+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:36.428 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:36.425+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:36.627 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:36.621+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:36.717 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:36.712+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:36.791 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:36.784+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:36.987 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:36.982+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:37.196 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:37.192+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:37.407 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:37.399+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:37.607 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:37.598+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:37.723 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:37.718+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:37.771 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:37.765+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:38.008 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:38.002+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:38.162 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:38.157+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:38.391 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:38.388+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:38.582 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:38.574+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:38.701 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:38.697+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:38.748 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:38.737+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:38.992 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:38.984+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:39.166 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:39.162+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:39.362 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:39.357+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:39.556 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:39.551+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:39.726 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:39.720+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:39.747 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:39.743+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:39.960 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:39.955+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:40.120 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:40.116+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:40.389 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:40.382+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:40.573 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:40.567+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:40.707 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:40.703+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:40.713 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:40.708+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:40.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~196s 2019-12-10T02:36:40.921 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:40.915+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:41.147 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:41.143+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:41.421 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:41.406+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:41.539 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:41.530+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:41.670 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:41.665+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:41.701 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:41.692+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:41.929 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:41.924+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:42.161 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:42.156+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:42.373 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:42.366+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:42.497 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:42.491+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:42.664 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:42.658+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:42.701 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:42.694+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:42.898 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:42.892+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:43.120 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:43.115+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:43.408 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:43.405+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:43.541 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:43.527+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:43.669 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:43.660+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:43.714 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:43.711+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:43.873 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:43.866+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:44.074 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:44.070+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:44.436 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:44.433+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:44.504 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:44.498+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:44.692 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:44.686+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:44.748 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:44.744+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:44.835 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:44.829+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:45.033 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:45.029+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:45.407 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:45.402+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:45.468 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:45.461+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:45.708 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:45.703+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:45.767 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:45.762+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:45.854 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:45.848+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:46.080 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:46.075+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:46.434 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:46.429+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:46.441 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:46.437+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:46.768 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:46.756+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:46.772 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:46.762+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:46.858 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:46.852+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:47.095 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:47.090+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:47.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~202s 2019-12-10T02:36:47.391 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:47.386+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:47.408 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:47.404+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:47.744 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:47.741+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:47.754 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:47.748+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:47.909 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:47.904+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:48.118 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:48.113+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:48.410 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:48.406+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:48.438 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:48.432+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:48.714 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:48.711+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:48.789 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:48.783+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:48.901 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:48.896+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:49.148 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:49.144+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:49.409 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:49.405+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:49.458 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:49.453+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:49.749 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:49.745+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:49.806 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:49.801+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:49.910 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:49.904+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:50.192 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:50.188+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:50.392 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:50.388+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:50.494 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:50.490+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:50.745 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:50.742+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:50.772 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:50.766+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:50.912 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:50.907+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:51.219 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:51.215+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:51.434 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:51.430+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:51.486 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:51.481+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:51.726 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:51.721+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:51.739 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:51.735+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:51.919 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:51.914+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:52.220 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:52.212+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:52.426 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:52.423+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:52.494 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:52.488+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:52.727 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:52.721+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:52.728 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:52.725+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:52.889 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:52.883+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:53.176 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:53.172+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:53.466 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:53.463+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:53.469 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~208s 2019-12-10T02:36:53.548 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:53.543+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:53.726 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:53.720+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:53.739 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:53.730+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:53.880 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:53.874+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:54.205 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:54.200+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:54.502 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:54.498+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:54.553 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:54.548+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:54.693 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:54.688+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:54.767 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:54.763+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:54.926 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:54.921+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:55.166 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:55.162+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:55.558 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:55.553+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:55.582 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:55.577+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:55.650 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:55.644+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:55.810 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:55.806+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:55.916 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:55.910+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:56.148 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:56.144+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:56.602 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:56.597+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:56.605 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:36:56.602+0000 1694f700 -1 osd.2 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.257306+0000 front 2019-12-10T02:35:44.258612+0000 (oldest deadline 2019-12-10T02:36:26.538897+0000) 2019-12-10T02:36:56.620 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:56.615+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:56.840 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:56.836+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:56.895 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:36:56.889+0000 1694f700 -1 osd.7 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:44.414559+0000 front 2019-12-10T02:35:44.548143+0000 (oldest deadline 2019-12-10T02:36:30.277143+0000) 2019-12-10T02:36:57.107 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:36:57.102+0000 1694f700 -1 osd.5 316 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:110115 2.2s0 2:40ce3d85:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.671_d77d2751e556447ca15aac1d801256d6:head [getxattrs out=1000b,stat out=16b,read 0~134] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e314) 2019-12-10T02:36:57.601 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:57.595+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:57.624 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:57.619+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:57.859 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:57.855+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:58.417 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:36:58.411+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:36:58.625 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:36:58.620+0000 1694f700 -1 osd.4 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:42.557335+0000 front 2019-12-10T02:35:42.512213+0000 (oldest deadline 2019-12-10T02:36:28.377223+0000) 2019-12-10T02:36:58.638 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:58.633+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:58.813 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:36:58.810+0000 1694f700 -1 osd.0 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.804642+0000 front 2019-12-10T02:35:43.604332+0000 (oldest deadline 2019-12-10T02:36:29.288411+0000) 2019-12-10T02:36:59.436 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:36:59.433+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:36:59.597 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:36:59.591+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:36:59.784 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~215s 2019-12-10T02:37:00.468 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:00.465+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:00.568 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:37:00.562+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:37:01.477 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:01.474+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:01.592 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:37:01.587+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:37:02.471 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:02.469+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:02.588 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:37:02.583+0000 1694f700 -1 osd.6 316 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:35:43.418489+0000 front 2019-12-10T02:35:43.796606+0000 (oldest deadline 2019-12-10T02:36:29.213725+0000) 2019-12-10T02:37:03.454 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:03.451+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:04.499 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:04.496+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:05.536 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:05.533+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:06.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~221s 2019-12-10T02:37:06.576 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:06.573+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:07.597 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:07.594+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:08.604 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:08.602+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:09.613 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:09.611+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:10.612 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:10.610+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:11.574 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:11.572+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:12.402 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~227s 2019-12-10T02:37:12.548 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:12.546+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:13.524 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:13.522+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:14.524 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:14.520+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:15.524 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:15.522+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:16.564 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:16.562+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:17.017 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:17.014+0000 35f8f700 -1 osd.1 316 build_incremental_map_msg missing incremental map 315 2019-12-10T02:37:17.019 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:17.015+0000 35f8f700 -1 osd.1 316 build_incremental_map_msg missing incremental map 316 2019-12-10T02:37:17.596 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:17.595+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:18.551 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:18.550+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:18.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~234s 2019-12-10T02:37:19.560 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:19.558+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:20.576 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:20.575+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:21.383 INFO:teuthology.orchestra.run.smithi067.stderr:testIfModifiedSince (test.functional.tests.TestFileComparison) ... ok 2019-12-10T02:37:21.560 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:21.558+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:22.615 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:22.606+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:23.631 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:23.628+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:24.677 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:24.674+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:25.022 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~240s 2019-12-10T02:37:25.663 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:25.660+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:26.716 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:26.713+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:27.677 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:27.672+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:28.652 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:28.648+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:29.623 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:29.620+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:30.589 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:30.587+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:31.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~246s 2019-12-10T02:37:31.625 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:31.622+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:32.593 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:32.591+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:33.623 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:33.620+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:34.635 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:34.633+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:35.618 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:35.615+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:36.573 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:36.571+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:37.592 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:37.589+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:37.648 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~252s 2019-12-10T02:37:38.589 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:38.586+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:39.553 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:39.545+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:40.576 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:40.574+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:41.542 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:41.540+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:42.529 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:42.526+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:43.572 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:43.570+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:43.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~259s 2019-12-10T02:37:44.590 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:44.587+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:44.657 INFO:teuthology.orchestra.run.smithi067.stderr:testIfNoneMatch (test.functional.tests.TestFileComparison) ... ok 2019-12-10T02:37:45.547 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:45.544+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:46.571 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:46.567+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:47.592 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:47.589+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:48.581 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:48.577+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:49.631 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:49.627+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:50.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~265s 2019-12-10T02:37:50.647 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:50.644+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:51.658 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:51.655+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:52.649 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:52.647+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:53.703 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:53.702+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:54.747 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:54.742+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:55.725 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:55.723+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:56.593 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~271s 2019-12-10T02:37:56.703 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:56.701+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:57.689 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:57.687+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:58.709 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:58.708+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:37:59.762 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:37:59.760+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:00.801 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:00.799+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:01.847 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:01.845+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:02.868 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:02.865+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:02.902 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~278s 2019-12-10T02:38:03.908 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:03.906+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:04.900 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:04.899+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:05.921 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:05.920+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:06.463 INFO:teuthology.orchestra.run.smithi067.stderr:testIfUnmodifiedSince (test.functional.tests.TestFileComparison) ... ok 2019-12-10T02:38:06.892 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:06.890+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:07.937 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:07.934+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:08.974 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:08.972+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:09.211 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~284s 2019-12-10T02:38:09.950 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:09.947+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:10.970 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:10.967+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:11.939 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:11.935+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:12.943 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:12.940+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:13.997 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:13.994+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:15.036 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:15.034+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:15.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~290s 2019-12-10T02:38:16.018 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:16.014+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:17.068 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:17.065+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:18.030 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:18.026+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:19.014 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:19.012+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:20.044 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:20.041+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:21.025 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:21.023+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:21.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~297s 2019-12-10T02:38:22.033 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:22.030+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:23.050 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:23.048+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:24.102 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:24.099+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:25.117 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:25.115+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:26.158 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:26.155+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:27.173 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:27.171+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:28.144 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~303s 2019-12-10T02:38:28.145 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2019-12-10T02:38:28.145 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-04-1433/qa/tasks/daemonwatchdog.py", line 38, in _run self.watch() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-04-1433/qa/tasks/daemonwatchdog.py", line 114, in watch self.bark() File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-abhi-testing-2019-12-04-1433/qa/tasks/daemonwatchdog.py", line 54, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/config.py", line 253, in __getattr__ raise AttributeError(name) AttributeError: mounts 2019-12-10T02:38:28.204 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:28.200+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:29.185 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:29.181+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:30.214 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:30.211+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:31.213 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:31.210+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:32.169 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:32.167+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:33.184 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:33.174+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:34.223 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:34.218+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:35.223 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:35.220+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:36.218 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:36.214+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:37.260 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:37.257+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:38.220 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:38.217+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:39.248 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:39.245+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:40.207 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:40.205+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:41.165 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:41.162+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:42.129 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:42.127+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:43.137 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:43.135+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:44.113 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:44.109+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:45.153 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:45.151+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:46.141 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:46.139+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:47.185 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:47.183+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:48.153 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:48.151+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:49.116 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:49.115+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:50.122 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:50.120+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:51.089 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:51.087+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:52.095 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:52.091+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:53.053 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:53.049+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:54.024 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:54.021+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:55.062 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:55.060+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:56.069 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:56.067+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:57.026 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:57.024+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:58.040 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:58.036+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:38:59.050 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:38:59.048+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:00.019 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:00.017+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:00.988 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:00.985+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:01.972 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:01.968+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:02.948 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:02.945+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:03.990 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:03.987+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:05.023 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:05.020+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:06.002 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:05.999+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:07.002 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:06.999+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:08.002 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:07.999+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:08.963 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:08.960+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:09.990 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:09.987+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:10.969 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:10.967+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:11.947 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:11.944+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:13.001 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:12.988+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:14.007 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:14.003+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:14.966 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:14.961+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:15.982 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:15.979+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:16.989 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:16.987+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:18.035 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:18.032+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:19.011 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:19.008+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:19.986 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:19.981+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:21.031 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:21.028+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:22.060 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:22.057+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:23.015 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:23.014+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:24.066 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:24.063+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:25.034 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:25.032+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:26.002 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:25.999+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:26.979 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:26.976+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:27.972 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:27.970+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:28.982 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:28.981+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:30.016 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:30.014+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:31.012 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:31.010+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:32.012 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:32.011+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:32.998 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:32.996+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:34.037 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:34.035+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:35.076 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:35.075+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:36.070 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:36.068+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:37.043 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:37.039+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:38.070 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:38.068+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:39.055 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:39.052+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:40.059 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:40.057+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:41.075 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:41.073+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:42.086 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:42.084+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:43.103 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:43.100+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:44.070 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:44.067+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:44.673 INFO:teuthology.orchestra.run.smithi067.stderr:testIfMatch (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-10T02:39:45.058 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:45.055+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:46.089 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:46.087+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:47.105 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:47.095+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:48.123 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:48.120+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:49.083 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:49.081+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:50.120 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:50.117+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:51.095 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:51.090+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:52.133 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:52.130+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:53.105 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:53.103+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:54.080 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:54.077+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:55.118 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:55.115+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:56.147 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:56.144+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:57.145 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:57.142+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:58.143 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:58.140+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:39:59.191 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:39:59.189+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:00.204 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:00.201+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:01.200 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:01.197+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:02.230 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:02.227+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:03.270 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:03.268+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:04.294 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:04.292+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:05.337 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:05.335+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:06.343 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:06.341+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:07.364 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:07.363+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:08.399 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:08.397+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:09.394 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:09.393+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:10.395 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:10.394+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:11.075 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:06:32.138 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:40:11.157 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:06:32.260 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:40:11.157 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:06:32.268 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:40:11.157 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:06:32.146 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:40:11.157 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:06:32.154 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:40:11.158 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:06:32.162 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:40:11.375 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:06:32.438 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:40:11.395 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:06:32.458 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:40:11.441 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:11.439+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:11.452 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:06:32.611 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:40:12.441 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:12.439+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:13.444 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:13.442+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:14.451 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:14.449+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:15.406 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:15.404+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:16.425 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:16.418+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:17.415 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:17.414+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:18.419 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:18.417+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:19.468 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:19.467+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:20.482 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:20.478+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:20.486 INFO:teuthology.orchestra.run.smithi067.stderr:testIfMatchAndUnmodified (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-10T02:40:21.444 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:21.443+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:22.490 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:22.486+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:23.460 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:23.455+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:24.492 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:24.489+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:25.464 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:25.460+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:26.502 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:26.498+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:27.480 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:27.477+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:28.499 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:28.496+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:29.470 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:29.467+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:30.493 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:30.490+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:31.516 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:31.513+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:32.530 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:32.528+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:33.493 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:33.490+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:34.472 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:34.469+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:35.503 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:35.500+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:36.476 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:36.473+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:37.442 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:37.440+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:38.414 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:38.411+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:39.421 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:39.419+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:40.395 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:40.392+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:41.431 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:41.428+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:41.635 INFO:teuthology.orchestra.run.smithi067.stderr:testIfModifiedSince (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-10T02:40:42.409 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:42.406+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:43.402 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:43.400+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:44.379 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:44.376+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:45.344 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:45.342+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:46.364 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:46.361+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:47.418 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:47.414+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:48.371 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:48.368+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:49.413 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:49.411+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:50.435 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:50.431+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:51.467 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:51.463+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:52.489 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:52.485+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:53.513 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:53.511+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:54.539 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:54.537+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:55.507 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:55.505+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:56.541 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:56.540+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:57.539 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:57.538+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:58.572 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:58.570+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:40:59.567 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:40:59.565+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:00.613 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:00.611+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:01.572 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:01.571+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:02.582 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:02.581+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:03.629 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:03.627+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:04.670 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:04.669+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:05.700 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:05.698+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:06.721 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:06.714+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:06.962 INFO:teuthology.orchestra.run.smithi067.stderr:testIfNoneMatch (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-10T02:41:07.691 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:07.687+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:08.738 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:08.732+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:09.769 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:09.765+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:10.749 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:10.746+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:11.802 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:11.799+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:12.832 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:12.828+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:13.858 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:13.855+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:14.818 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:14.814+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:15.810 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:15.808+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:16.767 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:16.764+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:17.767 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:17.764+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:18.742 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:18.739+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:19.734 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:19.731+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:20.753 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:20.748+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:21.780 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:21.777+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:22.746 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:22.743+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:23.754 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:23.750+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:24.711 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:24.709+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:25.728 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:25.726+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:26.754 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:26.751+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:27.710 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:27.707+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:28.667 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:28.663+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:29.698 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:29.695+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:30.750 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:30.746+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:31.257 INFO:teuthology.orchestra.run.smithi067.stderr:testIfUnmodifiedSince (test.functional.tests.TestFileComparisonUTF8) ... ok 2019-12-10T02:41:31.802 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:31.800+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:32.840 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:32.836+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:33.807 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:33.804+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:34.829 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:34.826+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:35.811 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:35.808+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:36.861 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:36.859+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:37.906 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:37.904+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:38.943 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:38.942+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:39.907 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:39.905+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:40.895 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:40.893+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:41.885 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:41.883+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:42.924 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:42.922+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:43.902 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:43.900+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:44.940 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:44.938+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:45.955 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:45.953+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:46.926 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:46.924+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:47.973 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:47.964+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:48.936 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:48.934+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:49.977 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:49.975+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:50.935 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:50.934+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:51.946 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:51.938+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:52.924 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:52.921+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:53.890 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:53.887+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:54.906 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:54.903+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:55.906 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:55.904+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:56.903 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:56.901+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:57.886 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:57.883+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:58.892 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:58.890+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:41:59.895 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:41:59.892+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:00.918 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:00.915+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:01.957 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:01.953+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:03.002 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:03.000+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:04.016 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:04.014+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:05.021 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:05.019+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:06.055 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:06.052+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:07.049 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:07.046+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:08.024 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:08.021+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:09.002 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:08.999+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:09.995 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:09.991+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:11.007 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:11.005+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:12.005 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:12.002+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:12.991 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:12.988+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:14.017 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:14.015+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:15.012 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:15.008+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:16.040 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:16.037+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:17.091 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:17.088+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:18.075 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:18.072+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:18.590 INFO:teuthology.orchestra.run.smithi067.stderr:testBadHeaders (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:42:18.659 INFO:teuthology.orchestra.run.smithi067.stderr:testBlankMetadataName (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:42:19.066 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:19.063+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:20.096 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:20.093+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:21.125 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:21.123+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:22.146 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:22.145+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:23.150 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:23.149+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:24.120 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:24.118+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:25.148 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:25.147+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:26.154 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:26.152+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:26.567 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:08:47.630 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:42:26.571 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:08:47.634 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:42:26.619 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:08:47.682 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:42:26.621 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:08:47.684 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:42:26.657 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:08:47.816 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:42:26.661 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:08:47.820 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:42:26.779 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:08:47.842 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:42:26.833 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:08:47.992 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:42:26.839 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:08:47.902 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:42:27.163 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:27.161+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:28.215 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:28.214+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:29.186 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:29.184+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:30.190 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:30.188+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:31.180 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:31.179+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:32.166 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:32.164+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:33.195 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:33.193+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:34.155 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:34.153+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:35.144 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:35.142+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:36.174 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:36.171+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:37.152 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:37.149+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:38.198 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:38.195+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:39.229 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:39.225+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:40.206 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:40.202+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:41.185 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:41.183+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:42.162 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:42.158+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:43.121 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:43.119+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:44.105 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:44.101+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:45.153 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:45.150+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:46.202 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:46.199+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:47.216 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:47.214+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:48.224 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:48.221+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:49.245 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:49.242+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:50.238 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:50.235+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:51.273 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:51.270+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:52.248 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:52.245+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:53.223 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:53.221+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:54.232 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:54.227+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:55.253 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:55.250+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:56.218 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:56.215+0000 1694f700 -1 osd.1 316 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:57.488 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:57.301+0000 1694f700 -1 osd.1 317 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:58.315 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:58.311+0000 1694f700 -1 osd.1 317 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:42:59.428 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:42:59.373+0000 1694f700 -1 osd.1 318 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:43:00.747 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:43:00.659+0000 1694f700 -1 osd.1 319 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:43:01.771 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:43:01.696+0000 1694f700 -1 osd.1 320 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:43:02.801 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:43:02.796+0000 1694f700 -1 osd.1 321 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4563.0:87150 7.2 7:4455f95b:lc::lc.27:head [call lock.lock in=50b] snapc 0=[] RETRY=44 ondisk+retry+write+known_if_redirected e315) 2019-12-10T02:43:36.958 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:09:58.021 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:43:36.964 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:09:58.027 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:43:36.976 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:09:58.135 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:43:36.980 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:09:58.140 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:43:36.982 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:09:58.045 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:43:36.989 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:09:58.050 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:43:37.163 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:09:58.322 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:43:37.169 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:09:58.228 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:43:37.242 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:09:58.305 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:45:42.328 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:42.321+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:42.383 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:42.378+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:42.992 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:42.983+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:43.322 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:43.317+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:43.424 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:43.416+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:44.019 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:44.014+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:44.306 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:44.301+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:44.389 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:44.384+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:45.011 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:45.007+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:45.072 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:45.067+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:45.297 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:45.291+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:45.431 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:45.420+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:45.740 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:45.734+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:45.745 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:45.742+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:46.027 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:46.023+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:46.108 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:46.100+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:46.337 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:46.330+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:46.472 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:46.464+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:46.719 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:46.713+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:46.759 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:46.754+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:46.986 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:46.981+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:47.092 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:47.087+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:47.365 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:47.360+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:47.483 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:47.471+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:47.687 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:47.683+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:47.732 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:47.717+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:47.994 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:47.991+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:48.062 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:48.056+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:48.414 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:48.409+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:48.513 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:48.507+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:48.694 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:48.689+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:48.733 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:48.728+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:49.005 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:49.000+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:49.057 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:49.052+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:49.426 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:49.422+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:49.514 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:49.507+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:49.688 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:49.684+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:49.726 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:49.721+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:50.002 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:49.997+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:50.023 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:50.017+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:50.425 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:50.416+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:50.475 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:50.468+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:50.698 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:50.692+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:50.716 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:50.710+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:51.044 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:51.039+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:51.052 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:51.048+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:51.455 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:51.447+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:51.480 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:51.475+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:51.677 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:51.670+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:51.730 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:51.724+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:52.068 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:52.064+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:52.083 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:52.078+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:52.461 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:52.456+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:52.495 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:52.490+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:52.727 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:52.721+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:52.758 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:52.755+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:53.079 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:53.074+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:53.080 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:53.073+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:53.418 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:53.412+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:53.503 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:53.498+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:53.726 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:53.721+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:53.736 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:53.731+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:54.047 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:54.043+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:54.101 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:54.095+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:54.450 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:54.443+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:54.526 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:54.522+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:54.703 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:54.699+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:54.708 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:54.706+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:55.072 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:55.068+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:55.150 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:55.143+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:55.442 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:55.437+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:55.552 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:55.547+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:55.701 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:55.697+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:55.743 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:55.740+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:56.068 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:56.063+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:56.122 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:56.116+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:56.435 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:56.430+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:56.581 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:56.571+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:56.672 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:56.667+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:56.733 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:56.729+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:57.072 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:57.065+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:57.170 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:57.165+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:57.436 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:57.431+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:57.559 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:57.554+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:57.682 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:57.678+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:57.764 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:57.760+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:58.088 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:58.085+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:58.173 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:58.168+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:58.425 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:58.420+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:58.545 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:58.538+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:58.682 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:58.678+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:58.800 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:58.796+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:45:59.105 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:45:59.101+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:45:59.208 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:45:59.202+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:45:59.436 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:45:59.430+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:45:59.589 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:45:59.583+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:45:59.653 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:45:59.648+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:45:59.780 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:45:59.775+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:00.143 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:00.139+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:00.183 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:00.178+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:00.417 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:00.411+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:00.641 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:00.637+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:00.653 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:00.648+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:00.829 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:00.825+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:01.180 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:01.176+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:01.225 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:01.220+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:01.420 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:01.414+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:01.629 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:01.624+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:01.692 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:01.687+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:01.785 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:01.782+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:02.135 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:02.130+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:02.268 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:02.263+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:02.434 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:02.429+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:02.664 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:02.658+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:02.687 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:02.682+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:02.748 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:02.744+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:03.094 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:03.090+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:03.289 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:03.283+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:03.422 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:03.416+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:03.663 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:03.658+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:03.669 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:03.663+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:03.770 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:03.767+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:04.077 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:04.074+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:04.326 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:04.319+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:04.464 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:04.459+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:04.632 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:04.627+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:04.667 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:04.663+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:04.789 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:04.786+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:05.048 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:05.044+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:05.368 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:05.363+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:05.465 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:05.459+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:05.621 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:05.617+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:05.715 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:05.710+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:05.806 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:05.804+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:06.031 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:06.029+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:06.371 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:06.366+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:06.475 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:06.468+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:06.590 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:06.584+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:06.748 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:06.744+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:06.812 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:06.808+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:06.997 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:06.993+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:07.369 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:07.364+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:07.461 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:07.456+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:07.614 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:07.609+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:07.766 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:07.760+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:07.797 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:07.790+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:07.976 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:07.973+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:08.412 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:08.400+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:08.491 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:08.485+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:08.588 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:08.582+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:08.762 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:08.759+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:08.778 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:08.773+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:08.975 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:08.972+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:09.412 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:09.408+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:09.538 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:09.532+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:09.624 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:09.619+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:09.752 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:09.748+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:09.804 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:09.801+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:09.970 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:09.967+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:10.445 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:10.439+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:10.533 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:10.527+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:10.615 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:10.605+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:10.765 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:10.760+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:10.780 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:10.776+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:10.934 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:10.931+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:11.492 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:11.487+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:11.582 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:11.576+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:11.604 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:11.598+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:11.735 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:11.733+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:11.741 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:11.736+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:11.940 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:11.937+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:12.365 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:46:12.364+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e360 v360) 2019-12-10T02:46:12.532 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:12.527+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:12.571 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:12.565+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:12.631 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:12.624+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:12.695 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:12.689+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:12.730 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:12.725+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:12.899 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:12.898+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:13.578 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:13.573+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:13.623 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:13.618+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:13.649 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:13.645+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:13.699 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:13.696+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:13.737 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:13.731+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:13.881 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:13.878+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:14.574 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:14.569+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:14.645 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:14.637+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:14.676 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:14.672+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:14.703 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:14.699+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:14.705 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:14.703+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:14.892 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:14.889+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:15.621 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:15.615+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:15.652 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:15.647+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:15.705 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:15.700+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:15.707 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:15.705+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:15.708 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:15.702+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:15.880 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:15.874+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:16.599 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:46:16.595+0000 1694f700 -1 osd.4 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.256684+0000 front 2019-12-10T02:45:01.248187+0000 (oldest deadline 2019-12-10T02:45:42.913015+0000) 2019-12-10T02:46:16.651 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:16.647+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:16.716 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:16.710+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:16.731 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:46:16.724+0000 1694f700 -1 osd.7 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:00.365212+0000 front 2019-12-10T02:45:00.360393+0000 (oldest deadline 2019-12-10T02:45:45.036205+0000) 2019-12-10T02:46:16.738 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:46:16.735+0000 1694f700 -1 osd.2 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.068137+0000 front 2019-12-10T02:45:01.181326+0000 (oldest deadline 2019-12-10T02:45:45.141460+0000) 2019-12-10T02:46:16.903 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:46:16.900+0000 1694f700 -1 osd.0 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:01.132111+0000 front 2019-12-10T02:45:00.978261+0000 (oldest deadline 2019-12-10T02:45:44.935836+0000) 2019-12-10T02:46:17.418 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:46:17.417+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 45sec e360 v360) 2019-12-10T02:46:17.648 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:46:17.644+0000 1694f700 -1 osd.6 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:45:02.095394+0000 front 2019-12-10T02:44:56.801648+0000 (oldest deadline 2019-12-10T02:45:41.885966+0000) 2019-12-10T02:46:17.761 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:46:17.756+0000 1694f700 -1 osd.5 360 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:44:57.031117+0000 front 2019-12-10T02:44:57.198631+0000 (oldest deadline 2019-12-10T02:45:42.280380+0000) 2019-12-10T02:46:52.962 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:13:14.025 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:46:52.971 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:13:14.034 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:46:53.061 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:13:14.124 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:46:53.065 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:13:14.225 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:46:53.068 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:13:14.228 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:46:53.069 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:13:14.132 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:46:53.347 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:13:14.410 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:46:53.360 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:13:14.424 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:46:53.396 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:13:14.555 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:39.406 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:14:00.469 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:39.415 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:14:00.478 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:39.424 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:14:00.487 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:39.428 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:14:00.491 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:39.433 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:14:00.593 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:39.449 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:14:00.608 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:39.482 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:14:00.545 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:39.490 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:14:00.553 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:39.538 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:14:00.698 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:47:41.605 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:41.599+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:42.461 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:42.456+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:42.600 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:42.594+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:43.076 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:43.073+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:43.350 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:43.344+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:43.463 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:43.461+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:43.561 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:43.556+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:44.035 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:44.031+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:44.317 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:44.311+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:44.491 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:44.489+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:44.604 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:44.599+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:45.049 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:45.046+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:45.297 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:45.292+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:45.484 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:45.478+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:45.496 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:45.492+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:45.562 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:45.556+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:46.091 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:46.087+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:46.317 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:46.312+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:46.452 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:46.449+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:46.479 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:46.474+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:46.605 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:46.594+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:47.108 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:47.105+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:47.322 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:47.316+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:47.415 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:47.413+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:47.511 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:47.505+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:47.599 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:47.593+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:48.080 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:48.077+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:48.374 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:48.366+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:48.408 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:48.404+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:48.534 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:48.529+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:48.572 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:48.567+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:49.095 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:49.090+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:49.412 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:49.407+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:49.452 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:49.447+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:49.494 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:49.489+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:49.607 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:49.602+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:50.068 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:50.061+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:50.411 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:50.408+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:50.436 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:50.430+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:50.460 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:50.455+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:50.574 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:50.569+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:51.084 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:51.080+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:51.388 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:51.385+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:51.436 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:51.431+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:51.461 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:51.456+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:51.582 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:51.577+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:52.063 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:52.059+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:52.427 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:52.421+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:52.429 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:52.425+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:52.447 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:52.441+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:52.573 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:52.569+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:53.045 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:53.042+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:53.439 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:53.435+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:53.455 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:53.450+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:53.485 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:53.480+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:53.597 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:53.591+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:54.042 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:54.039+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:54.407 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:54.402+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:54.472 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:54.466+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:54.473 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:54.466+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:54.603 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:54.598+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:55.064 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:55.059+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:55.397 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:55.393+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:55.438 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:55.433+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:55.493 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:55.487+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:55.637 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:55.629+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:56.036 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:56.032+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:56.416 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:56.412+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:56.450 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:56.445+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:56.485 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:56.480+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:56.669 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:56.664+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:57.047 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:57.041+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:57.378 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:57.374+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:57.450 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:57.446+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:57.484 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:57.480+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:57.715 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:57.709+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:58.072 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:58.068+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:58.355 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:58.351+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:58.424 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:58.418+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:58.478 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:58.472+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:58.689 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:58.685+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:47:59.113 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:47:59.109+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:47:59.399 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:47:59.395+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:47:59.460 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:47:59.454+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:47:59.461 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:47:59.456+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:47:59.677 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:47:59.672+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:00.163 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:00.160+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:00.431 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:00.428+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:00.473 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:00.468+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:00.505 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:00.500+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:00.642 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:00.635+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:01.149 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:01.139+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:01.422 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:01.415+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:01.444 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:01.436+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:01.531 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:01.526+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:01.616 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:01.600+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:02.099 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:02.093+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:02.470 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:02.467+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:02.478 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:02.469+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:02.515 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:02.509+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:02.597 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:02.593+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:03.100 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:03.095+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:03.492 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:03.487+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:03.524 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:03.518+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:03.540 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:03.535+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:03.581 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:03.576+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:04.054 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:04.050+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:04.491 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:04.487+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:04.525 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:04.520+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:04.539 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:04.534+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:04.540 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:04.534+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:05.072 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:05.067+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:05.502 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:05.496+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:05.570 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:05.564+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:05.571 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:05.566+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:05.577 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:05.572+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:06.125 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:06.120+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:06.537 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:06.529+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:06.545 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:06.540+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:06.548 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:06.544+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:06.596 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:06.590+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:07.127 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:07.121+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:07.524 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:07.514+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:07.545 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:07.542+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:07.549 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:07.544+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:07.566 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:07.560+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:08.163 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:08.158+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:08.509 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:08.505+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:08.528 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:08.523+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:08.565 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:08.560+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:08.588 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:08.584+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:09.135 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:09.130+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:09.486 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:09.480+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:09.502 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:09.497+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:09.549 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:09.544+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:09.624 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:09.619+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:10.127 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:10.123+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:10.458 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:48:10.452+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.083264+0000 front 2019-12-10T02:47:00.070778+0000 (oldest deadline 2019-12-10T02:48:10.323813+0000) 2019-12-10T02:48:10.518 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:10.514+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:10.542 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:10.531+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:10.593 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:10.588+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:10.652 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:10.645+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:11.116 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:11.112+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:11.422 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:48:11.416+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.083264+0000 front 2019-12-10T02:47:00.070778+0000 (oldest deadline 2019-12-10T02:48:10.323813+0000) 2019-12-10T02:48:11.564 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:11.559+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:11.564 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:11.559+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:11.598 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:11.592+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:11.649 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:11.644+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:12.095 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:48:12.091+0000 1694f700 -1 osd.2 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:57.958848+0000 front 2019-12-10T02:46:57.960149+0000 (oldest deadline 2019-12-10T02:47:42.793745+0000) 2019-12-10T02:48:12.474 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:48:12.467+0000 1694f700 -1 osd.7 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.083264+0000 front 2019-12-10T02:47:00.070778+0000 (oldest deadline 2019-12-10T02:48:10.323813+0000) 2019-12-10T02:48:12.560 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:48:12.555+0000 1694f700 -1 osd.0 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.584774+0000 front 2019-12-10T02:47:00.582732+0000 (oldest deadline 2019-12-10T02:47:41.924507+0000) 2019-12-10T02:48:12.588 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:48:12.584+0000 1694f700 -1 osd.4 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:47:00.503782+0000 front 2019-12-10T02:47:00.555249+0000 (oldest deadline 2019-12-10T02:47:42.435839+0000) 2019-12-10T02:48:12.650 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:48:12.645+0000 1694f700 -1 osd.6 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:58.486018+0000 front 2019-12-10T02:46:58.510216+0000 (oldest deadline 2019-12-10T02:47:41.351727+0000) 2019-12-10T02:48:12.680 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:48:12.675+0000 1694f700 -1 osd.5 371 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:46:59.444474+0000 front 2019-12-10T02:46:59.417168+0000 (oldest deadline 2019-12-10T02:47:45.251310+0000) 2019-12-10T02:48:54.227 INFO:teuthology.orchestra.run.smithi067.stderr:testChunkedPut (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:48:59.954 INFO:teuthology.orchestra.run.smithi067.stderr:testContentTypeGuessing (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:49:16.358 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:49:16.212+0000 35f8f700 -1 osd.5 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:16.362 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:49:16.217+0000 35f8f700 -1 osd.5 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:16.440 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:49:16.306+0000 36790700 -1 osd.7 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:16.504 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:49:16.335+0000 32f89700 -1 osd.5 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:16.553 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:49:16.358+0000 3578e700 -1 osd.7 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:16.574 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:49:16.347+0000 31786700 -1 osd.5 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:16.613 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:49:16.461+0000 30f85700 -1 osd.7 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:16.652 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:49:16.464+0000 30f85700 -1 osd.7 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:16.667 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:49:16.486+0000 34f8d700 -1 osd.7 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:16.790 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:49:16.660+0000 37792700 -1 osd.5 375 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:18.496 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:49:18.455+0000 37792700 -1 osd.7 377 build_incremental_map_msg missing incremental map 375 2019-12-10T02:49:19.210 INFO:teuthology.orchestra.run.smithi067.stderr:testCopy (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:49:25.277 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-10T02:49:25.237+0000 20e25700 -1 RGWWatcher::handle_error cookie 362636144 err (107) Transport endpoint is not connected 2019-12-10T02:49:46.106 INFO:teuthology.orchestra.run.smithi067.stderr:testCopy404s (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:49:48.479 INFO:teuthology.orchestra.run.smithi067.stderr:testCopyDestinationSlashProblems (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:50:13.500 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:13.494+0000 33f8b700 -1 osd.5 387 build_incremental_map_msg missing incremental map 375 2019-12-10T02:50:14.274 INFO:teuthology.orchestra.run.smithi067.stderr:testCopyFromHeader (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:50:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:testCopyFromHeader404s (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:50:21.448 INFO:teuthology.orchestra.run.smithi067.stderr:testCopyNoDestinationHeader (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:50:23.620 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:23.615+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:24.624 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:24.619+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:24.783 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:24.780+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:25.278 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:25.273+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:25.582 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:25.576+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:25.781 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:25.778+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:26.238 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:26.232+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:26.623 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:26.618+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:26.783 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:26.780+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:26.917 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:26.912+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:27.196 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:27.190+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:27.593 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:27.589+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:27.631 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:27.624+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:27.667 INFO:teuthology.orchestra.run.smithi067.stderr:testDelete (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:50:27.828 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:27.824+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:27.943 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:27.937+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:28.247 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:28.242+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:28.319 INFO:teuthology.orchestra.run.smithi067.stderr:testDeleteOfFileThatDoesNotExist (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:50:28.380 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:28.375+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:28.567 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:28.564+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:28.602 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:28.596+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:28.789 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:28.785+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:28.982 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:28.976+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:29.223 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:29.218+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:29.366 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:29.360+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:29.559 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:29.553+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:29.575 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:29.572+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:29.765 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:29.761+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:30.021 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:30.016+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:30.281 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:30.258+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:30.360 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:30.354+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:30.529 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:30.524+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:30.563 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:30.560+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:30.741 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:30.737+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:31.069 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:31.061+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:31.226 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:31.220+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:31.344 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:31.338+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:31.513 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:31.506+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:31.558 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:31.554+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:31.703 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:31.699+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:32.081 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:32.075+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:32.261 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:32.255+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:32.377 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:32.371+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:32.499 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:32.494+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:32.576 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:32.572+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:32.659 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:32.655+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:33.110 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:33.104+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:33.280 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:33.275+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:33.293 INFO:teuthology.orchestra.run.smithi067.stderr:testDeleteThen404s (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:50:33.350 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:33.344+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:33.532 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:33.528+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:33.568 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:33.563+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:33.650 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:33.647+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:34.076 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:34.071+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:34.248 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:34.243+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:34.358 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:34.352+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:34.571 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:34.566+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:34.585 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:34.582+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:34.631 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:34.629+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:35.064 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:35.059+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:35.303 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:35.298+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:35.342 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:35.336+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:35.594 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:35.588+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:35.613 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:35.609+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:35.669 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:35.666+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:36.109 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:36.104+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:36.328 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:36.323+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:36.344 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:36.338+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:36.597 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:36.594+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:36.605 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:36.600+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:36.690 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:36.687+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:37.104 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:37.099+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:37.353 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:37.349+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:37.354 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:37.349+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:37.580 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:37.577+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:37.660 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:37.655+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:37.683 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:37.681+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:38.086 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:38.081+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:38.310 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:38.306+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:38.384 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:38.380+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:38.613 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:38.610+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:38.667 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:38.664+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:38.670 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:38.664+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:39.051 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:39.046+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:39.365 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:39.358+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:39.372 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:39.362+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:39.584 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:39.582+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:39.625 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:39.623+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:39.691 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:39.685+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:40.096 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:40.091+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:40.326 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:40.316+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:40.400 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:40.395+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:40.585 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:40.580+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:40.608 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:40.605+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:40.653 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:40.647+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:41.124 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:41.118+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:41.349 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:41.341+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:41.390 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:41.384+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:41.599 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:41.596+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:41.635 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:41.632+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:41.673 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:41.668+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:42.156 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:42.151+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:42.358 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:42.352+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:42.395 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:42.390+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:42.602 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:42.600+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:42.626 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:42.623+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:42.681 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:42.676+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:43.172 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:43.166+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:43.395 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:43.389+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:43.457 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:43.442+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:43.589 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:43.586+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:43.602 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:43.599+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:43.720 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:43.715+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:44.224 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:44.207+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:44.409 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:44.404+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:44.497 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:44.489+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:44.556 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:44.552+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:44.605 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:44.603+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:44.740 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:44.736+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:45.201 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:45.196+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:45.457 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:45.450+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:45.532 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:45.527+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:45.539 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:45.535+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:45.606 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:45.603+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:45.702 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:45.696+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:46.187 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:46.182+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:46.498 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:46.492+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:46.564 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:46.559+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:46.567 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:46.564+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:46.568 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:46.565+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:46.683 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:46.678+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:47.237 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:47.232+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:47.538 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:47.532+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:47.539 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:47.533+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:47.571 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:47.565+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:47.609 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:47.606+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:47.658 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:47.652+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:48.226 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:48.220+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:48.516 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:48.512+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:48.560 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:48.555+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:48.565 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:48.559+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:48.644 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:48.641+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:48.661 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:48.655+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:48.778 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:17:09.841 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:50:48.813 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:17:09.875 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:50:48.821 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:17:09.884 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:50:48.831 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:17:09.991 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:50:48.836 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:17:09.899 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:50:48.848 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:17:10.007 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:50:48.998 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:17:10.061 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:50:49.032 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:17:10.192 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:50:49.146 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:17:10.209 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:50:49.202 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:49.198+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:49.521 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:49.515+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:49.536 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:49.526+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:49.578 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:49.574+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:49.618 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:49.615+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:49.669 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:49.663+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:50.223 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:50.217+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:50.550 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:50.546+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:50.555 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:50.551+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:50.604 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:50.601+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:50.617 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:50.613+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:50.664 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:50.659+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:51.247 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:51.241+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:51.562 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:51.557+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:51.581 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:51.578+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:51.582 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:51.578+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:51.640 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:51.637+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:51.691 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:51.686+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:52.233 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:52.222+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:52.531 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:52.525+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:52.562 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:52.557+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:52.626 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:52.622+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:52.671 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:52.668+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:52.737 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:52.732+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:53.257 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:53.250+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:53.578 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:53.573+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:53.602 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:53.596+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:53.610 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:53.606+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:53.628 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:53.624+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:53.749 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:53.743+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:54.264 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:54.259+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:54.561 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:54.553+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:54.572 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:54.565+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:54.601 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:54.598+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:54.615 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:54.612+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:54.761 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:54.751+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:55.244 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:55.238+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:55.521 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:55.515+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:55.596 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:55.592+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:55.597 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:55.592+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:55.649 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:55.645+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:55.732 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:55.726+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:56.283 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:56.277+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:56.560 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:56.556+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:56.588 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:56.583+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:56.633 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:56.626+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:56.645 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:56.639+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:56.713 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:56.703+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:57.313 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:57.305+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:57.571 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:57.564+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:57.581 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:57.577+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:57.606 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:57.601+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:57.640 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:57.625+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:57.709 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:57.704+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:58.319 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:58.314+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:58.555 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:58.548+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:58.600 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:58.595+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:58.609 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:58.605+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:58.668 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:58.664+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:58.693 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:58.687+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:50:58.775 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:50:58.771+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 4 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e387 v387) 2019-12-10T02:50:59.338 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:50:59.331+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:50:59.532 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:50:59.522+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:50:59.586 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:50:59.581+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:50:59.604 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:50:59.601+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:50:59.635 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:50:59.631+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:50:59.648 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:50:59.643+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:51:00.370 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:51:00.365+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:51:00.552 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:51:00.547+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:51:00.568 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:51:00.563+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:51:00.603 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:51:00.598+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:51:00.620 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:51:00.615+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:51:00.683 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:51:00.680+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:51:01.389 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:51:01.384+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:51:01.535 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:51:01.530+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:51:01.547 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:51:01.542+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:51:01.616 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:51:01.610+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:51:01.635 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:51:01.632+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:51:01.697 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:51:01.694+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:51:02.428 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:51:02.421+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:51:02.510 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:51:02.504+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:51:02.543 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:51:02.537+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:51:02.584 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:51:02.578+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:51:02.612 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:51:02.609+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:51:02.684 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:51:02.680+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:51:03.418 INFO:tasks.ceph.osd.7.smithi081.stderr:2019-12-10T02:51:03.413+0000 1694f700 -1 osd.7 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:41.494168+0000 front 2019-12-10T02:49:42.106168+0000 (oldest deadline 2019-12-10T02:50:26.153395+0000) 2019-12-10T02:51:03.475 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:51:03.469+0000 1694f700 -1 osd.5 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.966298+0000 front 2019-12-10T02:49:42.544319+0000 (oldest deadline 2019-12-10T02:50:27.338495+0000) 2019-12-10T02:51:03.549 INFO:tasks.ceph.osd.6.smithi081.stderr:2019-12-10T02:51:03.543+0000 1694f700 -1 osd.6 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.496445+0000 front 2019-12-10T02:49:42.481654+0000 (oldest deadline 2019-12-10T02:50:23.592067+0000) 2019-12-10T02:51:03.575 INFO:tasks.ceph.osd.2.smithi067.stderr:2019-12-10T02:51:03.572+0000 1694f700 -1 osd.2 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.960243+0000 front 2019-12-10T02:49:43.018104+0000 (oldest deadline 2019-12-10T02:50:24.012229+0000) 2019-12-10T02:51:03.596 INFO:tasks.ceph.osd.4.smithi081.stderr:2019-12-10T02:51:03.591+0000 1694f700 -1 osd.4 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:40.144661+0000 front 2019-12-10T02:49:40.147633+0000 (oldest deadline 2019-12-10T02:50:24.520205+0000) 2019-12-10T02:51:03.711 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T02:51:03.707+0000 1694f700 -1 osd.0 387 heartbeat_check: no reply from 172.21.15.67:6830 osd.1 since back 2019-12-10T02:49:42.567335+0000 front 2019-12-10T02:49:42.568631+0000 (oldest deadline 2019-12-10T02:50:26.746369+0000) 2019-12-10T02:51:03.851 INFO:tasks.ceph.mon.a.smithi067.stderr:2019-12-10T02:51:03.843+0000 1a89c700 -1 mon.a@0(leader) e1 get_health_metrics reporting 6 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:172.21.15.67:6826/10706,v1:172.21.15.67:6827/10706] for 46sec e387 v387) 2019-12-10T02:51:04.792 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:51:04.491+0000 1694f700 -1 osd.5 388 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:113460 2.2s0 2:4288a3ea:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.673_%eb%8c%8c%ed%88%90%e1%a0%81%e3%9c%83%e8%94%8a%e0%a4%81%e8%94%8b%ed%88%8e%e8%94%88%ed%88%90%e2%98%83%e9%90%89%e9%90%8b%e8%94%88%e0%a4%82:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~527 in=527b,setxattr user.rgw.manifest (387) in=404b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e387) 2019-12-10T02:51:05.638 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:51:05.542+0000 1694f700 -1 osd.5 389 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:113460 2.2s0 2:4288a3ea:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.673_%eb%8c%8c%ed%88%90%e1%a0%81%e3%9c%83%e8%94%8a%e0%a4%81%e8%94%8b%ed%88%8e%e8%94%88%ed%88%90%e2%98%83%e9%90%89%e9%90%8b%e8%94%88%e0%a4%82:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~527 in=527b,setxattr user.rgw.manifest (387) in=404b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e387) 2019-12-10T02:51:06.602 INFO:tasks.ceph.osd.5.smithi081.stderr:2019-12-10T02:51:06.594+0000 1694f700 -1 osd.5 389 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4557.0:113460 2.2s0 2:4288a3ea:::1ad16726-ce4c-4982-bd45-b63e0fe58407.4563.673_%eb%8c%8c%ed%88%90%e1%a0%81%e3%9c%83%e8%94%8a%e0%a4%81%e8%94%8b%ed%88%8e%e8%94%88%ed%88%90%e2%98%83%e9%90%89%e9%90%8b%e8%94%88%e0%a4%82:head [create,setxattr user.rgw.idtag (48) in=62b,setxattr user.rgw.tail_tag (48) in=65b,writefull 0~527 in=527b,setxattr user.rgw.manifest (387) in=404b,setxattr user.rgw.acl (193) in=205b,setxattr user.rgw.compression (55) in=75b,setxattr user.rgw.content_type (25) in=46b,setxattr user.rgw.etag (32) in=45b,call rgw.obj_store_pg_ver in=44b,setxattr user.rgw.source_zone (4) in=24b] snapc 0=[] ondisk+write+known_if_redirected e387) 2019-12-10T02:51:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:testEtagResponse (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:51:07.499 INFO:teuthology.orchestra.run.smithi067.stderr:testEtagWayoff (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:51:29.131 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:17:50.194 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:51:29.141 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:17:50.204 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:51:29.150 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:17:50.213 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:51:29.158 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:17:50.221 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:51:29.164 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:17:50.324 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:51:29.176 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:17:50.336 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:51:29.213 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:17:50.275 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:51:29.235 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:17:50.296 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:51:29.284 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:17:50.443 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:51:47.011 INFO:teuthology.orchestra.run.smithi067.stderr:testFileCreate (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:51:48.573 INFO:teuthology.orchestra.run.smithi067.stderr:testGetContentType (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:51:49.238 INFO:teuthology.orchestra.run.smithi067.stderr:testGetOnFileThatDoesNotExist (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:51:50.386 INFO:teuthology.orchestra.run.smithi067.stderr:testHead (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:51:51.451 INFO:teuthology.orchestra.run.smithi067.stderr:testHeadOnFileThatDoesNotExist (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:51:58.515 INFO:tasks.ceph.osd.1.smithi067.stderr:*** Caught signal (Aborted) ** 2019-12-10T02:51:58.516 INFO:tasks.ceph.osd.1.smithi067.stderr: in thread 32788700 thread_name:tp_osd_tp 2019-12-10T02:51:58.625 INFO:tasks.ceph.osd.1.smithi067.stderr: ceph version 15.0.0-7936-g827e03c (827e03c8a6b1e17d93dcaf79724437bac291c35b) octopus (dev) 2019-12-10T02:51:58.625 INFO:tasks.ceph.osd.1.smithi067.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-10T02:51:58.625 INFO:tasks.ceph.osd.1.smithi067.stderr: 2: (std::basic_ostream >& std::__ostream_insert >(std::basic_ostream >&, char const*, long)+0) [0xc9876e0] 2019-12-10T02:51:58.625 INFO:tasks.ceph.osd.1.smithi067.stderr: 3: (std::ostream& std::operator<< >(std::ostream&, std::vector > const&)+0x58) [0x71e878] 2019-12-10T02:51:58.626 INFO:tasks.ceph.osd.1.smithi067.stderr: 4: (operator<<(std::ostream&, PeeringState const&)+0x8b) [0x91feab] 2019-12-10T02:51:58.626 INFO:tasks.ceph.osd.1.smithi067.stderr: 5: (operator<<(std::ostream&, PG const&)+0x20) [0x76c870] 2019-12-10T02:51:58.626 INFO:tasks.ceph.osd.1.smithi067.stderr: 6: (PG::gen_prefix(std::ostream&) const+0x1ca) [0x76d05a] 2019-12-10T02:51:58.626 INFO:tasks.ceph.osd.1.smithi067.stderr: 7: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x1be) [0x81f93e] 2019-12-10T02:51:58.626 INFO:tasks.ceph.osd.1.smithi067.stderr: 8: (cls_cxx_read2(void*, int, int, ceph::buffer::v14_2_0::list*, unsigned int)+0x165) [0x691a75] 2019-12-10T02:51:58.626 INFO:tasks.ceph.osd.1.smithi067.stderr: 9: (()+0x22ffc) [0x205ebffc] 2019-12-10T02:51:58.626 INFO:tasks.ceph.osd.1.smithi067.stderr: 10: (()+0x15342) [0x205de342] 2019-12-10T02:51:58.627 INFO:tasks.ceph.osd.1.smithi067.stderr: 11: (()+0x65cdee) [0x764dee] 2019-12-10T02:51:58.627 INFO:tasks.ceph.osd.1.smithi067.stderr: 12: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&)+0x5a) [0x76517a] 2019-12-10T02:51:58.627 INFO:tasks.ceph.osd.1.smithi067.stderr: 13: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x3e73) [0x8235f3] 2019-12-10T02:51:58.627 INFO:tasks.ceph.osd.1.smithi067.stderr: 14: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x16f) [0x832a9f] 2019-12-10T02:51:58.627 INFO:tasks.ceph.osd.1.smithi067.stderr: 15: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3ac) [0x83323c] 2019-12-10T02:51:58.627 INFO:tasks.ceph.osd.1.smithi067.stderr: 16: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x837aaf] 2019-12-10T02:51:58.627 INFO:tasks.ceph.osd.1.smithi067.stderr: 17: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x83d8ad] 2019-12-10T02:51:58.628 INFO:tasks.ceph.osd.1.smithi067.stderr: 18: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x6dd619] 2019-12-10T02:51:58.628 INFO:tasks.ceph.osd.1.smithi067.stderr: 19: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x914fc2] 2019-12-10T02:51:58.628 INFO:tasks.ceph.osd.1.smithi067.stderr: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x945) [0x6f8075] 2019-12-10T02:51:58.628 INFO:tasks.ceph.osd.1.smithi067.stderr: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x45a) [0xcd228a] 2019-12-10T02:51:58.628 INFO:tasks.ceph.osd.1.smithi067.stderr: 22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xcda690] 2019-12-10T02:51:58.628 INFO:tasks.ceph.osd.1.smithi067.stderr: 23: (()+0x7e65) [0xc096e65] 2019-12-10T02:51:58.628 INFO:tasks.ceph.osd.1.smithi067.stderr: 24: (clone()+0x6d) [0xd20e88d] 2019-12-10T02:51:58.661 INFO:tasks.ceph.osd.1.smithi067.stderr:2019-12-10T02:51:58.657+0000 32788700 -1 *** Caught signal (Aborted) ** 2019-12-10T02:51:58.661 INFO:tasks.ceph.osd.1.smithi067.stderr: in thread 32788700 thread_name:tp_osd_tp 2019-12-10T02:51:58.661 INFO:tasks.ceph.osd.1.smithi067.stderr: 2019-12-10T02:51:58.662 INFO:tasks.ceph.osd.1.smithi067.stderr: ceph version 15.0.0-7936-g827e03c (827e03c8a6b1e17d93dcaf79724437bac291c35b) octopus (dev) 2019-12-10T02:51:58.662 INFO:tasks.ceph.osd.1.smithi067.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-10T02:51:58.662 INFO:tasks.ceph.osd.1.smithi067.stderr: 2: (std::basic_ostream >& std::__ostream_insert >(std::basic_ostream >&, char const*, long)+0) [0xc9876e0] 2019-12-10T02:51:58.662 INFO:tasks.ceph.osd.1.smithi067.stderr: 3: (std::ostream& std::operator<< >(std::ostream&, std::vector > const&)+0x58) [0x71e878] 2019-12-10T02:51:58.662 INFO:tasks.ceph.osd.1.smithi067.stderr: 4: (operator<<(std::ostream&, PeeringState const&)+0x8b) [0x91feab] 2019-12-10T02:51:58.662 INFO:tasks.ceph.osd.1.smithi067.stderr: 5: (operator<<(std::ostream&, PG const&)+0x20) [0x76c870] 2019-12-10T02:51:58.663 INFO:tasks.ceph.osd.1.smithi067.stderr: 6: (PG::gen_prefix(std::ostream&) const+0x1ca) [0x76d05a] 2019-12-10T02:51:58.663 INFO:tasks.ceph.osd.1.smithi067.stderr: 7: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x1be) [0x81f93e] 2019-12-10T02:51:58.663 INFO:tasks.ceph.osd.1.smithi067.stderr: 8: (cls_cxx_read2(void*, int, int, ceph::buffer::v14_2_0::list*, unsigned int)+0x165) [0x691a75] 2019-12-10T02:51:58.663 INFO:tasks.ceph.osd.1.smithi067.stderr: 9: (()+0x22ffc) [0x205ebffc] 2019-12-10T02:51:58.663 INFO:tasks.ceph.osd.1.smithi067.stderr: 10: (()+0x15342) [0x205de342] 2019-12-10T02:51:58.663 INFO:tasks.ceph.osd.1.smithi067.stderr: 11: (()+0x65cdee) [0x764dee] 2019-12-10T02:51:58.663 INFO:tasks.ceph.osd.1.smithi067.stderr: 12: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&)+0x5a) [0x76517a] 2019-12-10T02:51:58.664 INFO:tasks.ceph.osd.1.smithi067.stderr: 13: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x3e73) [0x8235f3] 2019-12-10T02:51:58.664 INFO:tasks.ceph.osd.1.smithi067.stderr: 14: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x16f) [0x832a9f] 2019-12-10T02:51:58.664 INFO:tasks.ceph.osd.1.smithi067.stderr: 15: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3ac) [0x83323c] 2019-12-10T02:51:58.664 INFO:tasks.ceph.osd.1.smithi067.stderr: 16: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x837aaf] 2019-12-10T02:51:58.664 INFO:tasks.ceph.osd.1.smithi067.stderr: 17: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x83d8ad] 2019-12-10T02:51:58.664 INFO:tasks.ceph.osd.1.smithi067.stderr: 18: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x6dd619] 2019-12-10T02:51:58.664 INFO:tasks.ceph.osd.1.smithi067.stderr: 19: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x914fc2] 2019-12-10T02:51:58.665 INFO:tasks.ceph.osd.1.smithi067.stderr: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x945) [0x6f8075] 2019-12-10T02:51:58.665 INFO:tasks.ceph.osd.1.smithi067.stderr: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x45a) [0xcd228a] 2019-12-10T02:51:58.665 INFO:tasks.ceph.osd.1.smithi067.stderr: 22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xcda690] 2019-12-10T02:51:58.665 INFO:tasks.ceph.osd.1.smithi067.stderr: 23: (()+0x7e65) [0xc096e65] 2019-12-10T02:51:58.665 INFO:tasks.ceph.osd.1.smithi067.stderr: 24: (clone()+0x6d) [0xd20e88d] 2019-12-10T02:51:58.665 INFO:tasks.ceph.osd.1.smithi067.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-10T02:51:58.665 INFO:tasks.ceph.osd.1.smithi067.stderr: 2019-12-10T02:51:59.024 INFO:tasks.ceph.osd.1.smithi067.stderr: 0> 2019-12-10T02:51:58.657+0000 32788700 -1 *** Caught signal (Aborted) ** 2019-12-10T02:51:59.024 INFO:tasks.ceph.osd.1.smithi067.stderr: in thread 32788700 thread_name:tp_osd_tp 2019-12-10T02:51:59.024 INFO:tasks.ceph.osd.1.smithi067.stderr: 2019-12-10T02:51:59.024 INFO:tasks.ceph.osd.1.smithi067.stderr: ceph version 15.0.0-7936-g827e03c (827e03c8a6b1e17d93dcaf79724437bac291c35b) octopus (dev) 2019-12-10T02:51:59.024 INFO:tasks.ceph.osd.1.smithi067.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-10T02:51:59.024 INFO:tasks.ceph.osd.1.smithi067.stderr: 2: (std::basic_ostream >& std::__ostream_insert >(std::basic_ostream >&, char const*, long)+0) [0xc9876e0] 2019-12-10T02:51:59.025 INFO:tasks.ceph.osd.1.smithi067.stderr: 3: (std::ostream& std::operator<< >(std::ostream&, std::vector > const&)+0x58) [0x71e878] 2019-12-10T02:51:59.025 INFO:tasks.ceph.osd.1.smithi067.stderr: 4: (operator<<(std::ostream&, PeeringState const&)+0x8b) [0x91feab] 2019-12-10T02:51:59.025 INFO:tasks.ceph.osd.1.smithi067.stderr: 5: (operator<<(std::ostream&, PG const&)+0x20) [0x76c870] 2019-12-10T02:51:59.025 INFO:tasks.ceph.osd.1.smithi067.stderr: 6: (PG::gen_prefix(std::ostream&) const+0x1ca) [0x76d05a] 2019-12-10T02:51:59.025 INFO:tasks.ceph.osd.1.smithi067.stderr: 7: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x1be) [0x81f93e] 2019-12-10T02:51:59.025 INFO:tasks.ceph.osd.1.smithi067.stderr: 8: (cls_cxx_read2(void*, int, int, ceph::buffer::v14_2_0::list*, unsigned int)+0x165) [0x691a75] 2019-12-10T02:51:59.025 INFO:tasks.ceph.osd.1.smithi067.stderr: 9: (()+0x22ffc) [0x205ebffc] 2019-12-10T02:51:59.026 INFO:tasks.ceph.osd.1.smithi067.stderr: 10: (()+0x15342) [0x205de342] 2019-12-10T02:51:59.026 INFO:tasks.ceph.osd.1.smithi067.stderr: 11: (()+0x65cdee) [0x764dee] 2019-12-10T02:51:59.026 INFO:tasks.ceph.osd.1.smithi067.stderr: 12: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&)+0x5a) [0x76517a] 2019-12-10T02:51:59.026 INFO:tasks.ceph.osd.1.smithi067.stderr: 13: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x3e73) [0x8235f3] 2019-12-10T02:51:59.026 INFO:tasks.ceph.osd.1.smithi067.stderr: 14: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x16f) [0x832a9f] 2019-12-10T02:51:59.026 INFO:tasks.ceph.osd.1.smithi067.stderr: 15: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3ac) [0x83323c] 2019-12-10T02:51:59.026 INFO:tasks.ceph.osd.1.smithi067.stderr: 16: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x837aaf] 2019-12-10T02:51:59.027 INFO:tasks.ceph.osd.1.smithi067.stderr: 17: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x83d8ad] 2019-12-10T02:51:59.027 INFO:tasks.ceph.osd.1.smithi067.stderr: 18: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x6dd619] 2019-12-10T02:51:59.027 INFO:tasks.ceph.osd.1.smithi067.stderr: 19: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x914fc2] 2019-12-10T02:51:59.027 INFO:tasks.ceph.osd.1.smithi067.stderr: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x945) [0x6f8075] 2019-12-10T02:51:59.027 INFO:tasks.ceph.osd.1.smithi067.stderr: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x45a) [0xcd228a] 2019-12-10T02:51:59.027 INFO:tasks.ceph.osd.1.smithi067.stderr: 22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xcda690] 2019-12-10T02:51:59.027 INFO:tasks.ceph.osd.1.smithi067.stderr: 23: (()+0x7e65) [0xc096e65] 2019-12-10T02:51:59.028 INFO:tasks.ceph.osd.1.smithi067.stderr: 24: (clone()+0x6d) [0xd20e88d] 2019-12-10T02:51:59.028 INFO:tasks.ceph.osd.1.smithi067.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-10T02:51:59.028 INFO:tasks.ceph.osd.1.smithi067.stderr: 2019-12-10T02:51:59.440 INFO:tasks.ceph.osd.1.smithi067.stderr: -7> 2019-12-10T02:51:58.657+0000 32788700 -1 *** Caught signal (Aborted) ** 2019-12-10T02:51:59.441 INFO:tasks.ceph.osd.1.smithi067.stderr: in thread 32788700 thread_name:tp_osd_tp 2019-12-10T02:51:59.441 INFO:tasks.ceph.osd.1.smithi067.stderr: 2019-12-10T02:51:59.441 INFO:tasks.ceph.osd.1.smithi067.stderr: ceph version 15.0.0-7936-g827e03c (827e03c8a6b1e17d93dcaf79724437bac291c35b) octopus (dev) 2019-12-10T02:51:59.441 INFO:tasks.ceph.osd.1.smithi067.stderr: 1: (()+0xf5f0) [0xc09e5f0] 2019-12-10T02:51:59.441 INFO:tasks.ceph.osd.1.smithi067.stderr: 2: (std::basic_ostream >& std::__ostream_insert >(std::basic_ostream >&, char const*, long)+0) [0xc9876e0] 2019-12-10T02:51:59.441 INFO:tasks.ceph.osd.1.smithi067.stderr: 3: (std::ostream& std::operator<< >(std::ostream&, std::vector > const&)+0x58) [0x71e878] 2019-12-10T02:51:59.442 INFO:tasks.ceph.osd.1.smithi067.stderr: 4: (operator<<(std::ostream&, PeeringState const&)+0x8b) [0x91feab] 2019-12-10T02:51:59.442 INFO:tasks.ceph.osd.1.smithi067.stderr: 5: (operator<<(std::ostream&, PG const&)+0x20) [0x76c870] 2019-12-10T02:51:59.442 INFO:tasks.ceph.osd.1.smithi067.stderr: 6: (PG::gen_prefix(std::ostream&) const+0x1ca) [0x76d05a] 2019-12-10T02:51:59.442 INFO:tasks.ceph.osd.1.smithi067.stderr: 7: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x1be) [0x81f93e] 2019-12-10T02:51:59.442 INFO:tasks.ceph.osd.1.smithi067.stderr: 8: (cls_cxx_read2(void*, int, int, ceph::buffer::v14_2_0::list*, unsigned int)+0x165) [0x691a75] 2019-12-10T02:51:59.442 INFO:tasks.ceph.osd.1.smithi067.stderr: 9: (()+0x22ffc) [0x205ebffc] 2019-12-10T02:51:59.442 INFO:tasks.ceph.osd.1.smithi067.stderr: 10: (()+0x15342) [0x205de342] 2019-12-10T02:51:59.443 INFO:tasks.ceph.osd.1.smithi067.stderr: 11: (()+0x65cdee) [0x764dee] 2019-12-10T02:51:59.443 INFO:tasks.ceph.osd.1.smithi067.stderr: 12: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v14_2_0::list&, ceph::buffer::v14_2_0::list&)+0x5a) [0x76517a] 2019-12-10T02:51:59.443 INFO:tasks.ceph.osd.1.smithi067.stderr: 13: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x3e73) [0x8235f3] 2019-12-10T02:51:59.443 INFO:tasks.ceph.osd.1.smithi067.stderr: 14: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x16f) [0x832a9f] 2019-12-10T02:51:59.443 INFO:tasks.ceph.osd.1.smithi067.stderr: 15: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x3ac) [0x83323c] 2019-12-10T02:51:59.444 INFO:tasks.ceph.osd.1.smithi067.stderr: 16: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x324f) [0x837aaf] 2019-12-10T02:51:59.444 INFO:tasks.ceph.osd.1.smithi067.stderr: 17: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0xc8d) [0x83d8ad] 2019-12-10T02:51:59.444 INFO:tasks.ceph.osd.1.smithi067.stderr: 18: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x2f9) [0x6dd619] 2019-12-10T02:51:59.444 INFO:tasks.ceph.osd.1.smithi067.stderr: 19: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x914fc2] 2019-12-10T02:51:59.444 INFO:tasks.ceph.osd.1.smithi067.stderr: 20: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x945) [0x6f8075] 2019-12-10T02:51:59.444 INFO:tasks.ceph.osd.1.smithi067.stderr: 21: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x45a) [0xcd228a] 2019-12-10T02:51:59.444 INFO:tasks.ceph.osd.1.smithi067.stderr: 22: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xcda690] 2019-12-10T02:51:59.445 INFO:tasks.ceph.osd.1.smithi067.stderr: 23: (()+0x7e65) [0xc096e65] 2019-12-10T02:51:59.445 INFO:tasks.ceph.osd.1.smithi067.stderr: 24: (clone()+0x6d) [0xd20e88d] 2019-12-10T02:51:59.445 INFO:tasks.ceph.osd.1.smithi067.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-10T02:51:59.445 INFO:tasks.ceph.osd.1.smithi067.stderr: 2019-12-10T02:52:02.703 INFO:tasks.ceph.osd.1.smithi067.stderr:daemon-helper: command crashed with signal 6 2019-12-10T02:52:09.628 INFO:teuthology.orchestra.run.smithi067.stderr:testMetadataOnPost (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:52:29.369 INFO:teuthology.orchestra.run.smithi067.stderr:testMetadataOnPut (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:52:37.279 INFO:teuthology.orchestra.run.smithi067.stderr:testNameLimit (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:52:37.438 INFO:teuthology.orchestra.run.smithi067.stderr:testNoContentLengthForPut (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:52:38.492 INFO:teuthology.orchestra.run.smithi067.stderr:testPostOnFileThatDoesNotExist (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:53:10.424 INFO:teuthology.orchestra.run.smithi067.stderr:testRangedGets (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:53:25.021 INFO:teuthology.orchestra.run.smithi067.stderr:testSerialization (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:54:20.027 INFO:teuthology.orchestra.run.smithi067.stderr:testStackedOverwrite (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:54:20.070 INFO:teuthology.orchestra.run.smithi067.stderr:testTooLongName (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:54:23.826 INFO:teuthology.orchestra.run.smithi067.stderr:testZeroByteFile (test.functional.tests.TestFileUTF8) ... ok 2019-12-10T02:55:55.198 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:22:16.260 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:55:55.207 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:22:16.270 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:55:55.223 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:22:16.383 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:55:55.226 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:22:16.289 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:55:55.231 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:22:16.294 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:55:55.235 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:22:16.395 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:55:55.502 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:22:16.565 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:55:55.515 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:22:16.577 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:55:55.516 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:22:16.675 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:57:24.013 INFO:teuthology.orchestra.run.smithi067.stderr:test_slo_etag_is_hash_of_etags (test.functional.tests.TestSlo) ... ok 2019-12-10T02:57:27.582 INFO:teuthology.orchestra.run.smithi067.stderr:test_slo_get_simple_manifest (test.functional.tests.TestSlo) ... ok 2019-12-10T02:57:29.793 INFO:teuthology.orchestra.run.smithi067.stderr:test_slo_ranged_get (test.functional.tests.TestSlo) ... ok 2019-12-10T02:58:09.257 INFO:teuthology.orchestra.run.smithi067.stderr:test_slo_etag_is_hash_of_etags (test.functional.tests.TestSloUTF8) ... ok 2019-12-10T02:58:12.896 INFO:teuthology.orchestra.run.smithi067.stderr:test_slo_get_simple_manifest (test.functional.tests.TestSloUTF8) ... ok 2019-12-10T02:58:15.521 INFO:teuthology.orchestra.run.smithi067.stderr:test_slo_ranged_get (test.functional.tests.TestSloUTF8) ... ok 2019-12-10T02:58:15.521 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-10T02:58:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:---------------------------------------------------------------------- 2019-12-10T02:58:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:Ran 147 tests in 7767.993s 2019-12-10T02:58:15.522 INFO:teuthology.orchestra.run.smithi067.stderr: 2019-12-10T02:58:15.522 INFO:teuthology.orchestra.run.smithi067.stderr:OK 2019-12-10T02:58:15.552 INFO:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 --cluster ceph user rm --uid foo.client.0 --purge-data 2019-12-10T02:58:15.704 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:15.703+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:15.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:15.735+0000 7fe4a246eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T02:58:15.736 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:15.735+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:15.744+0000 7fe4a246eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T02:58:15.748 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:15.744+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:15.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:15.755+0000 7fe4a246eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T02:58:15.756 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:15.755+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:15.787 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:15.786+0000 7fe4a246eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T02:58:16.026 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.026+0000 7fe4a246eec0 20 RGWRados::pool_iterate: got zone_info.1ad16726-ce4c-4982-bd45-b63e0fe58407 2019-12-10T02:58:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.026+0000 7fe4a246eec0 20 RGWRados::pool_iterate: got zonegroup_info.92ebed75-473d-4c06-bf7f-e5813082677e 2019-12-10T02:58:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.026+0000 7fe4a246eec0 20 RGWRados::pool_iterate: got zone_names.default 2019-12-10T02:58:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.026+0000 7fe4a246eec0 20 RGWRados::pool_iterate: got zonegroups_names.default 2019-12-10T02:58:16.027 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.026+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.051+0000 7fe4a246eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T02:58:16.052 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.051+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.064 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.063+0000 7fe4a246eec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T02:58:16.064 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.063+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.076+0000 7fe4a246eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T02:58:16.077 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.076+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.087+0000 7fe4a246eec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T02:58:16.088 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.087+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.093 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.093+0000 7fe4a246eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T02:58:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.093+0000 7fe4a246eec0 10 cannot find current period zonegroup using local zonegroup 2019-12-10T02:58:16.094 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.093+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.099+0000 7fe4a246eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T02:58:16.100 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.099+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.113+0000 7fe4a246eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T02:58:16.114 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.113+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.122+0000 7fe4a246eec0 20 rados_obj.operate() r=0 bl.length=382 2019-12-10T02:58:16.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.122+0000 7fe4a246eec0 20 zonegroup default 2019-12-10T02:58:16.123 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.122+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.128+0000 7fe4a246eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T02:58:16.128 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.128+0000 7fe4a246eec0 10 Cannot find current period zone using local zone 2019-12-10T02:58:16.129 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.128+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.138 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.137+0000 7fe4a246eec0 20 rados_obj.operate() r=-2 bl.length=0 2019-12-10T02:58:16.138 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.137+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.157+0000 7fe4a246eec0 20 rados_obj.operate() r=0 bl.length=46 2019-12-10T02:58:16.158 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.157+0000 7fe4a246eec0 20 rados->read ofs=0 len=0 2019-12-10T02:58:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.170+0000 7fe4a246eec0 20 rados_obj.operate() r=0 bl.length=951 2019-12-10T02:58:16.171 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.170+0000 7fe4a246eec0 20 zone default found 2019-12-10T02:58:16.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.170+0000 7fe4a246eec0 20 started sync module instance, tier type = 2019-12-10T02:58:16.172 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.170+0000 7fe4a246eec0 20 started zone id=1ad16726-ce4c-4982-bd45-b63e0fe58407 (name=default) with tier type = 2019-12-10T02:58:16.548 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.548+0000 7fe4a246eec0 20 add_watcher() i=0 2019-12-10T02:58:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.548+0000 7fe4a246eec0 20 add_watcher() i=1 2019-12-10T02:58:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.548+0000 7fe4a246eec0 20 add_watcher() i=2 2019-12-10T02:58:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.548+0000 7fe4a246eec0 20 add_watcher() i=3 2019-12-10T02:58:16.549 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.548+0000 7fe4a246eec0 20 add_watcher() i=4 2019-12-10T02:58:16.550 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.548+0000 7fe4a246eec0 20 add_watcher() i=5 2019-12-10T02:58:16.571 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.570+0000 7fe4a246eec0 20 add_watcher() i=6 2019-12-10T02:58:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.582+0000 7fe4a246eec0 20 add_watcher() i=7 2019-12-10T02:58:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.582+0000 7fe4a246eec0 2 all 8 watchers are set, enabling cache 2019-12-10T02:58:16.583 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.582+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T02:58:16.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.597+0000 7fe48f698700 0 -- 172.21.15.67:0/872097677 >> v2:172.21.15.67:3300/0 conn(0x55bc7a31e810 msgr2=0x55bc7a31ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T02:58:16.698 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:16.697+0000 7fe487fff700 0 -- 172.21.15.67:0/872097677 >> [v2:172.21.15.81:6812/9679,v1:172.21.15.81:6814/9679] conn(0x55bc7a336830 msgr2=0x55bc7a338c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T02:58:38.583 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:58:38.581+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T02:59:00.583 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:59:00.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T02:59:10.815 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:25:31.878 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:59:10.856 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:25:31.918 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:59:10.881 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:25:31.944 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:59:10.883 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:25:31.946 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:59:10.935 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:25:32.094 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:59:10.943 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:25:32.102 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:59:11.016 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:25:32.079 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T02:59:11.037 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:25:32.100 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T02:59:11.151 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:25:32.311 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T02:59:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:59:22.582+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T02:59:44.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T02:59:44.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:00:06.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:00:06.582+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:00:28.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:00:28.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:00:34.098 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:26:55.161 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:00:34.108 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:26:55.170 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:00:34.117 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:26:55.177 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:00:34.120 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:26:55.279 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:00:34.120 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:26:55.183 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:00:34.125 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:26:55.285 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:00:34.188 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:26:55.251 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:00:34.239 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:26:55.302 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:00:34.311 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:26:55.471 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:00:50.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:00:50.582+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:01:12.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:01:12.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:01:34.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:01:34.582+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:01:56.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:01:56.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:02:18.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:02:18.582+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:02:40.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:02:40.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:02:41.912 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:29:02.975 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:02:41.952 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:29:03.015 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:02:41.975 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:29:03.038 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:02:42.004 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:29:03.067 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:02:42.029 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:29:03.188 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:02:42.035 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:29:03.195 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:02:42.237 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:29:03.300 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:02:42.249 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:29:03.312 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:02:42.308 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:29:03.468 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:03:02.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:03:02.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:03:24.584 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:03:24.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:03:46.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:03:46.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:04:08.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:04:08.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:04:19.440 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:30:40.502 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:04:19.452 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:30:40.514 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:04:19.457 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:30:40.519 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:04:19.458 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:30:40.617 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:04:19.466 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:30:40.529 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:04:19.477 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:30:40.636 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:04:19.555 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:30:40.617 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:04:19.562 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:30:40.625 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:04:19.588 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:30:40.747 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:04:30.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:04:30.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:04:52.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:04:52.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:05:14.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:05:14.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:05:36.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:05:36.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:05:58.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:05:58.583+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:06:20.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:06:20.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:06:42.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:06:42.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:07:04.586 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:07:04.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:07:26.585 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:07:26.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:07:33.060 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:33:54.122 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:07:33.063 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:33:54.126 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:07:33.115 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:33:54.176 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:07:33.117 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:33:54.180 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:07:33.120 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:33:54.279 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:07:33.140 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:33:54.300 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:07:33.173 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:33:54.236 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:07:33.223 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:33:54.286 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:07:33.279 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:33:54.439 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:07:48.586 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:07:48.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:08:10.586 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:08:10.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:08:32.586 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:08:32.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:08:54.586 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:08:54.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:09:16.586 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:09:16.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:09:28.620 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:35:49.683 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:09:28.623 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:35:49.686 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:09:28.641 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:35:49.801 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:09:28.644 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:35:49.707 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:09:28.652 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:35:49.715 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:09:28.664 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:35:49.824 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:09:28.731 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:35:49.794 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:09:28.775 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:35:49.831 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:09:28.849 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:35:50.009 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:09:38.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:09:38.584+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:10:00.586 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:10:00.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:10:22.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:10:22.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:10:44.586 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:10:44.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:11:06.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:11:06.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:11:28.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:11:28.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:11:50.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:11:50.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:12:12.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:12:12.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:12:34.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:12:34.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:12:56.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:12:56.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:13:18.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:13:18.585+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:13:40.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:13:40.586+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:14:02.587 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:14:02.586+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:14:24.588 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:14:24.586+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:14:41.832 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:41:02.895 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:14:41.842 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:41:02.905 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:14:41.851 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:41:02.914 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:14:41.857 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:41:02.920 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:14:41.865 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:41:03.025 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:14:41.874 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:41:03.034 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:14:42.077 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:41:03.140 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:14:42.078 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:41:03.141 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:14:42.182 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:41:03.342 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:14:46.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:14:46.586+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:15:08.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:15:08.586+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:15:30.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:15:30.586+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:15:52.588 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:15:52.586+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:15:54.974 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:42:16.037 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:15:55.026 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:42:16.089 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:15:55.042 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:42:16.202 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:15:55.046 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:42:16.109 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:15:55.050 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:42:16.210 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:15:55.051 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:42:16.113 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:15:55.076 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:42:16.139 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:15:55.106 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:42:16.169 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:15:55.137 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:42:16.297 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:16:14.588 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:16:14.587+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:16:36.588 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:16:36.586+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:16:58.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:16:58.587+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:17:20.588 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:17:20.587+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:17:42.590 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:17:42.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:18:04.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:18:04.587+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:18:26.588 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:18:26.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:18:48.588 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:18:48.587+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:19:10.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:19:10.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:19:32.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:19:32.587+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:19:54.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:19:54.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:19:55.048 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:46:16.111 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:19:55.057 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:46:16.121 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:19:55.063 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:46:16.126 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:19:55.067 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:46:16.131 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:19:55.070 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:46:16.230 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:19:55.084 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:46:16.244 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:19:55.218 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:46:16.281 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:19:55.307 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:46:16.467 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:19:55.311 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:46:16.374 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:20:16.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:20:16.587+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:20:38.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:20:38.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:21:00.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:21:00.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:21:22.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:21:22.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:21:44.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:21:44.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:22:06.589 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:22:06.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:22:29.071 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:22:28.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:22:50.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:22:50.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:23:12.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:23:12.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:23:34.726 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:23:34.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:23:56.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:23:56.588+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:24:16.764 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:50:37.737 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:24:16.890 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:50:37.759 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:24:16.892 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:50:37.791 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:24:16.892 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:50:37.774 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:24:16.931 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:50:37.938 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:24:16.934 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:50:37.944 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:24:16.937 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:50:37.897 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:24:16.937 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:50:37.919 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:24:16.968 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:50:38.127 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:24:18.590 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:24:18.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:24:40.590 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:24:40.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:24:41.048 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:24:41.015+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe470031450 msgr2=0x7fe470033980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T03:24:55.716 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:24:55.709+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe470037710 msgr2=0x7fe470039b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T03:24:56.532 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:24:56.000+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6812/9679,v1:172.21.15.81:6814/9679] conn(0x7fe4700224d0 msgr2=0x7fe47003efe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T03:25:02.912 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:25:02.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:25:05.237 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:51:26.299 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:25:05.241 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:51:26.303 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:25:05.260 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:51:26.321 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:25:05.263 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:51:26.326 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:25:05.267 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:51:26.427 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:25:05.292 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:51:26.355 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:25:05.305 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:51:26.464 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:25:05.328 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:51:26.391 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:25:05.427 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:51:26.584 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:25:24.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:25:24.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:25:46.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:25:46.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:26:08.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:26:08.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:26:30.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:26:30.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:26:52.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:26:52.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:27:14.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:27:14.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:27:36.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:27:36.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:27:41.060 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:27:41.058+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe470031420 msgr2=0x7fe470033980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T03:27:58.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:27:58.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:28:20.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:28:20.589+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:28:42.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:28:42.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:29:04.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:29:04.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:29:26.591 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:29:26.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:29:48.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:29:48.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:30:10.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:30:10.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:30:18.183 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:56:39.245 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:30:18.193 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:56:39.252 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:30:18.201 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:56:39.264 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:30:18.209 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:56:39.271 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:30:18.214 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:56:39.374 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:30:18.226 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:56:39.385 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:30:18.428 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:56:39.491 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:30:18.457 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:56:39.520 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:30:18.478 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:56:39.638 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:30:32.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:30:32.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:30:54.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:30:54.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:31:16.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:31:16.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:31:38.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:31:38.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:32:00.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:32:00.590+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:32:22.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:32:22.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:32:38.661 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:58:59.724 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:32:38.666 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:58:59.729 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:32:38.710 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:58:59.773 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:32:38.711 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:58:59.774 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:32:38.752 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:58:59.910 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:32:38.776 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:58:59.912 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:32:38.787 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:06:58:59.849 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:32:38.791 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:06:58:59.854 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:32:38.904 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:06:59:00.063 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:32:44.592 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:32:44.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:33:06.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:33:06.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:33:28.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:33:28.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:33:50.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:33:50.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:34:12.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:34:12.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:34:34.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:34:34.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:34:56.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:34:56.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:35:18.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:35:18.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:35:26.677 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:01:47.737 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:35:26.684 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:01:47.741 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:35:26.700 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:01:47.754 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:35:26.701 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:01:47.759 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:35:26.706 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:01:47.861 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:35:26.711 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:01:47.868 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:35:26.812 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:01:47.875 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:35:26.821 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:01:47.884 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:35:26.871 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:01:48.031 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:35:40.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:35:40.591+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:36:02.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:36:02.592+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:36:24.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:36:24.592+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:36:46.593 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:36:46.592+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:37:08.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:37:08.592+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:37:30.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:37:30.592+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:37:52.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:37:52.592+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:38:14.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:38:14.592+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:38:36.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:38:36.593+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:38:58.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:38:58.592+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:39:20.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:39:20.593+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:39:42.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:39:42.593+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:40:04.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:40:04.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:40:26.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:40:26.593+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:40:40.510 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:07:01.573 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:40:40.520 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:07:01.583 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:40:40.529 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:07:01.592 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:40:40.539 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:07:01.599 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:40:40.540 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:07:01.700 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:40:40.547 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:07:01.707 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:40:40.748 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:07:01.810 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:40:40.752 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:07:01.912 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:40:40.818 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:07:01.880 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:40:48.594 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:40:48.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:41:00.650 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:07:21.713 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:41:00.655 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:07:21.718 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:41:00.703 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:07:21.765 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:41:00.704 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:07:21.767 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:41:00.709 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:07:21.772 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:41:00.738 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:07:21.898 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:41:00.742 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:07:21.805 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:41:00.745 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:07:21.905 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:41:00.785 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:07:21.945 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:41:10.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:41:10.593+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:41:32.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:41:32.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:41:54.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:41:54.593+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:42:16.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:42:16.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:42:38.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:42:38.593+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:43:00.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:43:00.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:43:22.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:43:22.593+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:43:36.389 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:09:57.452 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:43:36.436 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:09:57.499 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:43:36.454 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:09:57.517 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:43:36.483 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:09:57.643 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:43:36.485 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:09:57.645 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:43:36.490 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:09:57.553 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:43:36.580 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:09:57.642 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:43:36.623 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:09:57.686 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:43:36.697 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:09:57.856 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:43:41.282 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T03:43:41.264+0000 21852700 -1 reset not still connected to 0x3aa245a0 2019-12-10T03:43:41.284 INFO:tasks.ceph.osd.0.smithi067.stderr:2019-12-10T03:43:41.265+0000 21852700 -1 reset not still connected to 0x3aeb18f0 2019-12-10T03:43:44.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:43:44.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:43:46.206 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:43:46.187+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe47000d900 msgr2=0x7fe47000dd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T03:44:06.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:44:06.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:44:28.595 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:44:28.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:44:50.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:44:50.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:45:12.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:45:12.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:45:34.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:45:34.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:45:51.097 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:12:12.160 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:45:51.108 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:12:12.171 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:45:51.121 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:12:12.184 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:45:51.125 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:12:12.284 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:45:51.130 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:12:12.193 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:45:51.141 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:12:12.301 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:45:51.255 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:12:12.318 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:45:51.259 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:12:12.323 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:45:51.321 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:12:12.481 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:45:56.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:45:56.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:46:18.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:46:18.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:46:40.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:46:40.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:47:02.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:47:02.594+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:47:24.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:47:24.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:47:46.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:47:46.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:48:08.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:48:08.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:48:30.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:48:30.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:48:52.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:48:52.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:49:14.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:49:14.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:49:22.794 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:15:43.857 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:49:22.828 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:15:43.891 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:49:22.858 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:15:43.921 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:49:22.861 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:15:43.924 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:49:22.883 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:15:44.043 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:49:22.885 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:15:44.045 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:49:23.009 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:15:44.170 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:49:23.014 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:15:44.077 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:49:23.035 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:15:44.098 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:49:36.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:49:36.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:49:58.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:49:58.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:50:20.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:50:20.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:50:42.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:50:42.595+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:51:01.224 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:17:22.287 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:51:01.235 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:17:22.298 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:51:01.237 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:17:22.300 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:51:01.241 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:17:22.304 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:51:01.253 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:17:22.413 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:51:01.266 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:17:22.427 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:51:01.340 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:17:22.402 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:51:01.358 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:17:22.421 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:51:01.396 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:17:22.556 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:51:04.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:51:04.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:51:26.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:51:26.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:51:48.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:51:48.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:52:10.597 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:52:10.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:52:32.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:52:32.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:52:54.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:52:54.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:53:16.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:53:16.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:53:38.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:53:38.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:54:00.600 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:54:00.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:54:22.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:54:22.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:54:44.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:54:44.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:55:06.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:55:06.596+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:55:28.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:55:28.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:55:50.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:55:50.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:56:12.598 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:56:12.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:56:14.395 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:22:35.458 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:56:14.405 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:22:35.468 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:56:14.410 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:22:35.473 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:56:14.414 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:22:35.477 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:56:14.433 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:22:35.593 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:56:14.456 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:22:35.616 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:56:14.623 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:22:35.686 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:56:14.628 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:22:35.691 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:56:14.753 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:22:35.913 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:56:34.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:56:34.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:56:56.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:56:56.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:57:18.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:57:18.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:57:40.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:57:40.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:57:44.687 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:24:05.750 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:57:44.691 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:24:05.754 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:57:44.730 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:24:05.890 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:57:44.749 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:24:05.812 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:57:44.751 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:24:05.814 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:57:44.761 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:24:05.922 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:57:44.774 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:24:05.837 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T03:57:44.811 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:24:05.874 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T03:57:44.851 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:24:06.011 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T03:57:45.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:57:45.735+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47003a070 msgr2=0x7fe470010d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T03:58:02.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:58:02.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:58:24.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:58:24.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:58:46.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:58:46.597+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:59:08.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:59:08.598+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:59:30.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:59:30.598+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T03:59:52.599 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T03:59:52.598+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:00:14.600 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:00:14.598+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:00:36.600 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:00:36.598+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:00:58.600 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:00:58.599+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:01:20.600 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:01:20.598+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:01:21.414 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:01:21.408+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x7fe47000d900 msgr2=0x7fe47000dd10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T04:01:23.559 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:27:44.622 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:01:23.567 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:27:44.630 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:01:23.588 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:27:44.748 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:01:23.588 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:27:44.651 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:01:23.594 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:27:44.657 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:01:23.596 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:27:44.757 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:01:23.789 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:27:44.950 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:01:23.809 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:27:44.872 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:01:23.851 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:27:44.914 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:01:42.600 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:01:42.599+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:02:04.600 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:02:04.598+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:02:26.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:02:26.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:02:48.600 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:02:48.599+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:03:10.600 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:03:10.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:03:32.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:03:32.599+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:03:54.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:03:54.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:04:16.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:04:16.599+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:04:38.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:04:38.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:05:00.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:05:00.599+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:05:22.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:05:22.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:05:44.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:05:44.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:06:06.582 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:32:27.645 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:06.593 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:32:27.656 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:06.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:06:06.601+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:06:06.617 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:32:27.680 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:06.619 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:32:27.682 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:06.635 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:32:27.796 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:06.641 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:32:27.802 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:06.746 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:32:27.809 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:06.784 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:32:27.847 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:06.791 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:32:27.952 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:28.601 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:06:28.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:06:35.923 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:32:56.985 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:35.931 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:32:56.994 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:35.943 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:32:57.006 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:35.949 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:32:57.012 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:35.952 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:32:57.112 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:35.973 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:32:57.036 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:35.975 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:32:57.135 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:36.004 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:32:57.067 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:36.095 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:32:57.255 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:06:50.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:06:50.601+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:07:12.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:07:12.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:07:34.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:07:34.601+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:07:46.469 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:07:46.465+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6802/10705,v1:172.21.15.67:6803/10705] conn(0x7fe488008750 msgr2=0x7fe488009ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T04:07:56.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:07:56.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:08:18.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:08:18.601+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:08:40.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:08:40.600+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:09:02.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:09:02.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:09:24.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:09:24.601+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:09:46.602 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:09:46.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:10:08.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:10:08.601+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:10:30.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:10:30.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:10:52.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:10:52.601+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:11:14.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:11:14.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:11:36.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:11:36.601+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:11:47.921 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:38:08.984 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:11:47.930 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:38:08.993 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:11:47.931 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:38:08.994 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:11:47.937 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:38:09.000 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:11:47.942 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:38:09.103 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:11:47.956 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:38:09.117 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:11:48.129 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:38:09.192 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:11:48.147 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:38:09.210 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:11:48.161 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:38:09.322 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:11:58.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:11:58.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:12:20.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:12:20.601+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:12:42.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:12:42.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:13:04.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:13:04.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:13:26.603 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:13:26.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:13:48.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:13:48.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:14:10.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:14:10.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:14:28.404 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:40:49.467 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:14:28.428 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:40:49.491 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:14:28.449 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:40:49.512 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:14:28.450 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:40:49.513 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:14:28.476 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:40:49.636 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:14:28.479 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:40:49.639 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:14:28.525 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:40:49.588 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:14:28.543 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:40:49.606 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:14:28.610 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:40:49.771 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:14:32.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:14:32.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:14:54.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:14:54.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:15:16.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:15:16.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:15:38.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:15:38.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:16:00.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:16:00.602+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:16:22.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:16:22.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:16:44.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:16:44.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:17:00.127 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:43:21.190 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:17:00.135 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:43:21.198 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:17:00.148 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:43:21.211 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:17:00.151 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:43:21.312 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:17:00.153 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:43:21.216 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:17:00.161 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:43:21.322 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:17:00.267 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:43:21.329 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:17:00.275 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:43:21.338 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:17:00.326 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:43:21.487 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:17:06.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:17:06.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:17:28.604 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:17:28.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:17:50.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:17:50.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:18:12.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:18:12.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:18:34.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:18:34.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:18:56.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:18:56.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:19:18.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:19:18.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:19:40.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:19:40.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:20:02.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:20:02.603+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:20:24.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:20:24.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:20:46.605 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:20:46.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:20:56.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:20:56.626+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe4700118d0 msgr2=0x7fe470042210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T04:21:08.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:21:08.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:21:30.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:21:30.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:21:52.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:21:52.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:22:12.354 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:48:33.416 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:12.363 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:48:33.426 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:12.371 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:48:33.434 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:12.375 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:48:33.438 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:12.383 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:48:33.543 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:12.390 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:48:33.550 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:12.580 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:48:33.643 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:12.619 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:48:33.682 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:12.724 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:48:33.885 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:14.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:22:14.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:22:36.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:22:36.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:22:55.212 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:49:16.275 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:55.234 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:49:16.297 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:55.239 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:49:16.302 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:55.248 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:49:16.311 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:55.284 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:49:16.445 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:55.291 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:49:16.452 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:55.304 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:49:16.366 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:55.340 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:49:16.500 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:55.342 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:49:16.405 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:22:58.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:22:58.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:23:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:23:20.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:23:42.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:23:42.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:24:04.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:24:04.604+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:24:15.807 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:50:36.870 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:24:15.851 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:50:36.913 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:24:15.900 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:50:36.963 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:24:15.902 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:50:36.965 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:24:15.918 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:50:37.079 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:24:15.926 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:50:37.086 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:24:15.927 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:50:36.989 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:24:15.981 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:50:37.044 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:24:16.036 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:50:37.196 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:24:26.606 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:24:26.605+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:24:48.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:24:48.605+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:25:10.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:25:10.605+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:25:32.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:25:32.605+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:25:54.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:25:54.605+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:26:16.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:26:16.605+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:26:38.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:26:38.605+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:27:00.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:27:00.606+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:27:21.507 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:53:42.570 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:27:21.515 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:53:42.578 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:27:21.527 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:53:42.590 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:27:21.531 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:53:42.594 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:27:21.533 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:53:42.694 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:27:21.537 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:53:42.698 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:27:21.653 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:53:42.716 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:27:21.660 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:53:42.723 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:27:21.745 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:53:42.906 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:27:22.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:27:22.605+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:27:44.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:27:44.606+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:28:06.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:28:06.606+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:28:28.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:28:28.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:28:50.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:28:50.606+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:29:12.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:29:12.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:29:34.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:29:34.606+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:29:56.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:29:56.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:30:18.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:30:18.606+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:30:40.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:30:40.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:31:02.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:31:02.606+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:31:17.266 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:57:38.328 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:31:17.289 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:57:38.352 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:31:17.316 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:57:38.379 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:31:17.319 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:57:38.382 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:31:17.338 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:57:38.498 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:31:17.344 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:57:38.505 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:31:17.410 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:57:38.473 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:31:17.484 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:57:38.547 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:31:17.535 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:57:38.696 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:31:24.608 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:31:24.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:31:46.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:31:46.606+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:32:08.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:32:08.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:32:30.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:32:30.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:32:32.640 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:58:53.702 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:32:32.644 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:58:53.707 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:32:32.663 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:58:53.725 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:32:32.666 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:58:53.826 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:32:32.668 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:58:53.731 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:32:32.671 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:58:53.832 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:32:32.737 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:07:58:53.800 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:32:32.752 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:07:58:53.815 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:32:32.820 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:07:58:53.981 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:32:52.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:32:52.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:33:14.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:33:14.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:33:36.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:33:36.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:33:58.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:33:58.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:34:20.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:34:20.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:34:42.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:34:42.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:35:04.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:35:04.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:35:26.609 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:35:26.607+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:35:48.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:35:48.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:36:10.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:36:10.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:36:32.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:36:32.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:36:54.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:36:54.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:37:16.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:37:16.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:37:38.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:37:38.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:37:43.635 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:04:04.698 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:37:43.646 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:04:04.708 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:37:43.659 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:04:04.721 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:37:43.664 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:04:04.726 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:37:43.669 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:04:04.829 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:37:43.679 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:04:04.840 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:37:43.869 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:04:04.931 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:37:43.871 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:04:04.934 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:37:43.892 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:04:05.052 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:38:00.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:38:00.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:38:22.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:38:22.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:38:44.610 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:38:44.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:39:06.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:39:06.608+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:39:28.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:39:28.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:39:39.147 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:06:00.209 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:39:39.200 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:06:00.262 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:39:39.229 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:06:00.389 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:39:39.234 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:06:00.395 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:39:39.238 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:06:00.301 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:39:39.240 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:06:00.302 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:39:39.265 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:06:00.328 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:39:39.319 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:06:00.381 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:39:39.360 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:06:00.521 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:39:50.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:39:50.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:40:12.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:40:12.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:40:34.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:40:34.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:40:56.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:40:56.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:41:18.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:41:18.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:41:40.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:41:40.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:42:02.611 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:42:02.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:42:24.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:42:24.609+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:42:46.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:42:46.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:42:54.881 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:09:15.943 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:42:54.886 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:09:15.948 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:42:54.896 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:09:15.959 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:42:54.904 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:09:15.967 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:42:54.910 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:09:16.071 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:42:54.921 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:09:16.079 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:42:55.038 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:09:16.100 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:42:55.043 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:09:16.106 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:42:55.076 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:09:16.237 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:43:08.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:43:08.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:43:30.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:43:30.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:43:52.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:43:52.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:44:14.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:44:14.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:44:36.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:44:36.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:44:58.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:44:58.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:45:20.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:45:20.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:45:42.612 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:45:42.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:46:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:46:04.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:46:26.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:46:26.610+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:46:48.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:46:48.611+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:47:10.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:47:10.611+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:47:32.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:47:32.611+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:47:54.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:47:54.612+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:48:00.986 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:14:22.049 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:00.992 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:14:22.054 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:01.039 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:14:22.102 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:01.041 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:14:22.104 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:01.063 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:14:22.223 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:01.066 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:14:22.227 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:01.182 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:14:22.244 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:01.188 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:14:22.250 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:01.243 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:14:22.404 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:06.021 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:14:27.083 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:06.029 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:14:27.091 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:06.041 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:14:27.103 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:06.046 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:14:27.108 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:06.048 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:14:27.209 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:06.058 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:14:27.219 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:06.059 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:14:27.122 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:06.071 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:14:27.133 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:06.132 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:14:27.292 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:48:16.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:48:16.611+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:48:38.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:48:38.612+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:49:00.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:49:00.611+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:49:22.613 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:49:22.612+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:49:31.921 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:49:31.917+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe47000a8b0 msgr2=0x7fe470049b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T04:49:44.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:49:44.611+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:50:06.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:50:06.613+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:50:28.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:50:28.612+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:50:50.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:50:50.613+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:51:12.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:51:12.612+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:51:34.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:51:34.613+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:51:56.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:51:56.612+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:52:18.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:52:18.613+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:52:40.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:52:40.612+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:53:02.614 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:53:02.613+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:53:18.205 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:19:39.268 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:53:18.209 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:19:39.272 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:53:18.219 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:19:39.282 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:53:18.225 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:19:39.287 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:53:18.227 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:19:39.387 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:53:18.241 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:19:39.402 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:53:18.423 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:19:39.486 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:53:18.441 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:19:39.503 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:53:18.509 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:19:39.670 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:53:24.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:53:24.612+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:53:46.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:53:46.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:54:08.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:54:08.613+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:54:30.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:54:30.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:54:52.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:54:52.613+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:55:14.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:55:14.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:55:36.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:55:36.613+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:55:58.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:55:58.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:56:20.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:56:20.613+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:56:22.908 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:22:43.971 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:56:22.957 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:22:44.019 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:56:22.966 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:22:44.029 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:56:23.009 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:22:44.170 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:56:23.013 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:22:44.174 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:56:23.022 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:22:44.085 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:56:23.057 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:22:44.120 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:56:23.143 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:22:44.304 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:56:23.163 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:22:44.226 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:56:42.615 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:56:42.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:57:04.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:57:04.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:57:26.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:57:26.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:57:48.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:57:48.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:58:10.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:58:10.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:58:29.888 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:24:50.951 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:58:29.894 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:24:50.956 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:58:29.907 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:24:50.969 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:58:29.909 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:24:50.971 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:58:29.918 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:24:51.079 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:58:29.926 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:24:51.087 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:58:29.995 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:24:51.058 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T04:58:30.005 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:24:51.068 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T04:58:30.090 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:24:51.250 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T04:58:32.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:58:32.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:58:54.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:58:54.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:59:16.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:59:16.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T04:59:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T04:59:38.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:00:00.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:00:00.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:00:22.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:00:22.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:00:44.616 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:00:44.614+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:01:06.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:01:06.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:01:28.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:01:28.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:01:50.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:01:50.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:02:12.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:02:12.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:02:34.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:02:34.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:02:56.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:02:56.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:03:18.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:03:18.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:03:40.443 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:30:01.505 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:03:40.453 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:30:01.516 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:03:40.463 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:30:01.526 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:03:40.469 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:30:01.532 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:03:40.471 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:30:01.632 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:03:40.475 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:30:01.636 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:03:40.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:03:40.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:03:40.668 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:30:01.730 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:03:40.711 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:30:01.773 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:03:40.730 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:30:01.891 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:04:02.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:04:24.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:04:24.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:04:44.635 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:31:05.697 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:44.682 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:31:05.745 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:44.694 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:31:05.756 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:44.695 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:31:05.758 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:44.703 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:31:05.864 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:44.705 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:31:05.866 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:44.741 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:31:05.804 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:44.748 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:31:05.810 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:44.775 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:31:05.936 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:04:46.617 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:04:46.615+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:05:08.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:05:08.616+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:05:10.079 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:31:31.141 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:05:10.091 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:31:31.154 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:05:10.131 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:31:31.193 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:05:10.135 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:31:31.197 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:05:10.138 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:31:31.201 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:05:10.173 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:31:31.235 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:05:10.192 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:31:31.353 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:05:10.197 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:31:31.358 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:05:10.244 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:31:31.404 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:05:30.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:05:30.616+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:05:52.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:05:52.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:06:14.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:06:14.616+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:06:36.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:06:36.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:06:58.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:06:58.616+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:07:12.079 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:07:12.076+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe47004a0b0 msgr2=0x7fe470010440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T05:07:20.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:07:20.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:07:42.618 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:07:42.616+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:08:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:08:04.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:08:26.619 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:08:26.616+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:08:48.619 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:08:48.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:08:50.191 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:35:11.254 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:08:50.208 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:35:11.270 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:08:50.214 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:35:11.277 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:08:50.221 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:35:11.283 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:08:50.222 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:35:11.383 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:08:50.233 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:35:11.394 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:08:50.374 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:35:11.437 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:08:50.377 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:35:11.439 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:08:50.456 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:35:11.617 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:09:10.619 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:09:10.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:09:32.619 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:09:32.618+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:09:54.619 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:09:54.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:10:02.145 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:10:02.142+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe470010980 msgr2=0x7fe47000d900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:10:16.619 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:10:16.618+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:10:38.619 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:10:38.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:11:00.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:11:00.618+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:11:22.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:11:22.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:11:44.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:11:44.618+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:12:06.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:12:06.617+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:12:28.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:12:28.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:12:50.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:12:50.618+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:13:06.834 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:39:27.896 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:13:06.877 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:39:27.939 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:13:06.878 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:39:27.941 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:13:06.885 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:39:27.948 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:13:06.899 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:39:28.060 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:13:06.922 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:39:28.083 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:13:06.989 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:39:28.051 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:13:07.058 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:39:28.219 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:13:07.117 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:39:28.179 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:13:12.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:13:12.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:13:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:13:34.618+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:13:56.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:13:56.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:14:02.100 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:40:23.162 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:14:02.107 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:40:23.169 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:14:02.117 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:40:23.179 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:14:02.121 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:40:23.184 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:14:02.132 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:40:23.292 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:14:02.141 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:40:23.302 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:14:02.177 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:40:23.239 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:14:02.191 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:40:23.254 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:14:02.246 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:40:23.407 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:14:18.620 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:14:18.618+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:14:40.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:14:40.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:15:02.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:15:02.618+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:15:24.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:15:24.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:15:46.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:15:46.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:16:08.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:16:08.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:16:30.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:16:30.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:16:52.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:16:52.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:17:14.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:17:14.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:17:36.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:17:36.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:17:58.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:17:58.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:18:20.621 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:18:20.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:18:42.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:18:42.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:19:04.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:19:04.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:19:13.283 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:45:34.345 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:19:13.294 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:45:34.355 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:19:13.304 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:45:34.367 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:19:13.307 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:45:34.370 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:19:13.321 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:45:34.481 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:19:13.337 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:45:34.498 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:19:13.497 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:45:34.560 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:19:13.520 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:45:34.582 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:19:13.600 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:45:34.761 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:19:26.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:19:26.619+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:19:48.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:19:48.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:20:10.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:20:10.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:20:32.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:20:32.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:20:54.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:20:54.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:21:16.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:21:16.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:21:28.736 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:47:49.798 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:21:28.785 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:47:49.847 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:21:28.787 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:47:49.849 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:21:28.788 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:47:49.850 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:21:28.816 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:47:49.977 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:21:28.819 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:47:49.980 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:21:28.867 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:47:49.924 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:21:28.882 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:47:49.944 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:21:28.970 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:47:50.130 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:21:38.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:21:38.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:22:00.622 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:22:00.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:22:22.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:22:22.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:22:44.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:22:44.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:23:06.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:23:06.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:23:28.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:23:28.620+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:23:50.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:23:50.621+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:24:12.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:24:12.621+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:24:25.497 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:50:46.559 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:24:25.506 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:50:46.568 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:24:25.516 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:50:46.578 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:24:25.519 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:50:46.680 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:24:25.521 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:50:46.584 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:24:25.530 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:50:46.691 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:24:25.636 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:50:46.698 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:24:25.646 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:50:46.709 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:24:25.688 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:50:46.849 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:24:34.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:24:34.622+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:24:56.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:24:56.621+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:25:18.623 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:25:18.622+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:25:40.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:25:40.621+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:26:02.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:26:02.622+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:26:24.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:26:24.621+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:26:46.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:26:46.622+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:27:08.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:27:08.622+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:27:30.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:27:30.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:27:52.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:27:52.622+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:28:14.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:28:14.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:28:36.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:28:36.622+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:28:58.624 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:28:58.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:29:20.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:29:20.622+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:29:36.126 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:55:57.188 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:36.136 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:55:57.199 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:36.150 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:55:57.213 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:36.158 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:55:57.220 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:36.163 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:55:57.324 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:36.185 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:55:57.346 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:36.348 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:55:57.410 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:36.400 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:55:57.462 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:36.443 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:55:57.603 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:42.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:29:42.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:29:50.547 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:56:11.609 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:50.601 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:56:11.663 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:50.622 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:08:56:11.685 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:50.658 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:56:11.721 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:50.662 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:56:11.724 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:50.671 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:56:11.832 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:50.675 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:56:11.836 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:50.681 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:08:56:11.743 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:29:50.703 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:08:56:11.864 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:30:04.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:30:04.622+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:30:26.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:30:26.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:30:48.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:30:48.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:31:10.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:31:10.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:31:32.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:31:32.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:31:54.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:31:54.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:32:16.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:32:16.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:32:38.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:32:38.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:33:00.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:33:00.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:33:22.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:33:22.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:33:44.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:33:44.623+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:34:06.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:34:06.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:34:22.407 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:34:22.404+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe470031420 msgr2=0x7fe470033980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:34:28.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:34:28.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:34:46.846 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:01:07.908 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:34:46.855 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:01:07.917 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:34:46.865 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:01:07.927 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:34:46.871 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:01:07.933 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:34:46.874 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:01:08.035 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:34:46.905 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:01:08.066 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:34:47.059 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:01:08.121 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:34:47.063 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:01:08.125 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:34:47.214 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:01:08.374 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:34:50.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:34:50.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:35:02.393 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:35:02.391+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x7fe4700118d0 msgr2=0x7fe470042210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:35:12.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:35:12.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:35:34.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:35:34.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:35:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:35:42.415+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe470034df0 msgr2=0x7fe47004d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:35:56.626 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:35:56.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:36:18.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:36:18.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:36:40.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:36:40.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:37:02.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:37:02.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:37:24.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:37:24.624+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:37:46.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:37:46.625+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:38:08.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:38:08.625+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:38:12.541 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:04:33.603 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:38:12.596 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:04:33.658 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:38:12.604 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:04:33.666 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:38:12.620 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:04:33.683 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:38:12.635 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:04:33.792 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:38:12.638 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:04:33.799 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:38:12.723 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:04:33.785 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:38:12.735 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:04:33.797 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:38:12.758 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:04:33.919 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:38:30.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:38:30.625+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:38:52.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:38:52.625+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:39:14.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:39:14.625+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:39:36.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:39:36.625+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:39:58.087 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:06:19.149 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:39:58.096 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:06:19.158 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:39:58.097 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:06:19.160 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:39:58.101 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:06:19.163 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:39:58.115 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:06:19.275 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:39:58.120 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:06:19.281 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:39:58.192 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:06:19.254 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:39:58.206 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:06:19.268 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:39:58.283 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:06:19.443 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:39:58.627 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:39:58.625+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:40:20.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:40:20.626+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:40:42.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:40:42.625+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:41:04.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:41:04.626+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:41:26.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:41:26.625+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:41:48.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:41:48.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:42:02.485 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:42:02.482+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe470034df0 msgr2=0x7fe47004d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:42:10.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:42:10.626+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:42:32.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:42:32.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:42:54.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:42:54.626+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:43:16.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:43:16.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:43:38.628 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:43:38.626+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:44:00.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:44:00.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:44:22.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:44:22.626+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:44:44.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:44:44.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:45:06.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:45:06.626+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:45:10.224 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:11:31.286 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:10.230 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:11:31.292 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:10.241 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:11:31.303 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:10.245 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:11:31.307 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:10.250 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:11:31.411 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:10.255 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:11:31.416 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:10.457 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:11:31.519 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:10.457 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:11:31.520 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:10.589 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:11:31.750 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:28.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:45:28.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:45:49.537 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:12:10.599 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:49.583 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:12:10.645 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:49.602 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:12:10.762 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:49.637 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:12:10.700 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:49.645 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:12:10.708 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:49.664 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:12:10.825 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:49.687 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:12:10.749 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:49.730 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:12:10.792 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:49.747 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:12:10.908 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:45:50.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:45:50.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:46:12.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:46:12.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:46:34.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:46:34.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:46:34.779 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:12:55.842 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:46:34.786 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:12:55.849 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:46:34.826 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:12:55.888 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:46:34.830 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:12:55.892 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:46:34.834 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:12:55.896 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:46:34.845 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:12:56.005 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:46:34.847 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:12:56.008 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:46:34.881 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:12:55.944 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:46:34.974 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:12:56.135 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:46:56.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:46:56.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:47:02.540 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:47:02.538+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe4700118d0 msgr2=0x7fe470042210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:47:18.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:47:18.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:47:40.629 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:47:40.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:47:45.822 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:47:45.820+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe47000b0b0 msgr2=0x7fe47004a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:48:02.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:48:02.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:48:24.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:48:24.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:48:46.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:48:46.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:49:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:49:08.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:49:30.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:49:30.627+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:49:52.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:49:52.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:50:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:50:14.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:50:20.260 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:16:41.322 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:50:20.267 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:16:41.330 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:50:20.281 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:16:41.343 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:50:20.287 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:16:41.349 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:50:20.291 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:16:41.451 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:50:20.304 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:16:41.465 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:50:20.448 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:16:41.511 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:50:20.478 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:16:41.639 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:50:20.494 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:16:41.557 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:50:36.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:50:36.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:50:58.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:50:58.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:51:20.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:51:20.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:51:42.630 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:51:42.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:52:04.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:52:04.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:52:26.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:52:26.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:52:48.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:52:48.628+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:53:10.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:53:10.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:53:32.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:53:32.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:53:54.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:53:54.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:54:16.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:54:16.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:54:38.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:54:38.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:54:56.879 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:21:17.941 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:54:56.926 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:21:17.989 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:54:56.958 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:21:18.020 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:54:56.974 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:21:18.135 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:54:56.980 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:21:18.043 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:54:56.983 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:21:18.144 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:54:57.109 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:21:18.171 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:54:57.149 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:21:18.211 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:54:57.197 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:21:18.358 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:00.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:55:00.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:55:22.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:55:22.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:55:32.189 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:21:53.251 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:32.201 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:21:53.263 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:32.207 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:21:53.268 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:32.212 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:21:53.275 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:32.218 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:21:53.379 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:32.226 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:21:53.386 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:32.260 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:21:53.322 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:32.290 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:21:53.352 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:32.339 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:21:53.499 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T05:55:44.631 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:55:44.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:55:47.596 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:55:47.594+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6802/10705,v1:172.21.15.67:6803/10705] conn(0x7fe470010980 msgr2=0x7fe47000d900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T05:56:06.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:56:06.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:56:28.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:56:28.629+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:56:50.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:56:50.630+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:57:12.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:57:12.630+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:57:34.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:57:34.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:57:56.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:57:56.630+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:58:06.738 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:58:06.733+0000 7fe48fe99700 0 -- 172.21.15.67:0/872097677 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x55bc7a325810 msgr2=0x55bc7a327d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T05:58:06.739 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:58:06.736+0000 7fe48fe99700 0 -- 172.21.15.67:0/872097677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe3d0031cb0 msgr2=0x7fe3d003eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:58:18.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:58:18.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:58:40.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:58:40.630+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:59:02.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:59:02.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:59:02.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:59:02.661+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe470009100 msgr2=0x7fe470049b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:59:24.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:59:24.630+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T05:59:27.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:59:27.661+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe47000de40 msgr2=0x7fe4700348b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T05:59:46.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T05:59:46.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:00:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:00:08.630+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:00:30.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:00:30.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:00:43.076 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:27:04.138 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:00:43.082 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:27:04.144 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:00:43.090 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:27:04.152 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:00:43.094 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:27:04.157 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:00:43.095 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:27:04.256 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:00:43.113 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:27:04.274 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:00:43.298 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:27:04.361 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:00:43.302 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:27:04.365 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:00:43.318 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:27:04.479 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:00:52.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:00:52.630+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:01:14.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:01:14.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:01:36.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:01:36.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:01:58.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:01:58.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:02:20.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:02:20.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:02:42.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:02:42.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:03:04.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:03:04.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:03:18.688 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:29:39.750 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:03:18.704 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:29:39.766 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:03:18.732 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:29:39.794 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:03:18.743 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:29:39.805 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:03:18.776 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:29:39.935 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:03:18.776 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:29:39.937 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:03:18.825 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:29:39.887 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:03:18.865 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:29:39.927 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:03:18.903 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:29:40.064 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:03:26.633 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:03:26.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:03:48.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:03:48.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:04:10.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:04:10.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:04:32.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:04:32.631+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:04:54.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:04:54.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:05:16.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:05:16.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:05:38.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:05:38.633+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:05:52.730 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:05:52.728+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe47000de40 msgr2=0x7fe4700348b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T06:05:55.353 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:32:16.413 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:05:55.362 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:32:16.425 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:05:55.376 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:32:16.439 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:05:55.381 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:32:16.444 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:05:55.392 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:32:16.552 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:05:55.405 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:32:16.566 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:05:55.505 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:32:16.567 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:05:55.530 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:32:16.593 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:05:55.612 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:32:16.773 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:06:00.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:06:00.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:06:22.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:06:22.633+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:06:44.634 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:06:44.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:07:06.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:07:06.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:07:28.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:07:28.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:07:50.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:07:50.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:08:12.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:08:12.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:08:34.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:08:34.632+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:08:56.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:08:56.633+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:09:18.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:09:18.633+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:09:40.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:09:40.633+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:10:02.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:10:02.633+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:10:24.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:10:24.633+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:10:46.635 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:10:46.633+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:10:47.398 INFO:tasks.rgw.client.0.smithi067.stdout:*** Caught signal (Segmentation fault) ** 2019-12-10T06:10:47.400 INFO:tasks.rgw.client.0.smithi067.stdout: in thread 18a15700 thread_name:msgr-worker-1 2019-12-10T06:10:47.471 INFO:tasks.rgw.client.0.smithi067.stdout: ceph version 15.0.0-7936-g827e03c (827e03c8a6b1e17d93dcaf79724437bac291c35b) octopus (dev) 2019-12-10T06:10:47.471 INFO:tasks.rgw.client.0.smithi067.stdout: 1: (()+0xf5f0) [0x100905f0] 2019-12-10T06:10:47.473 INFO:tasks.rgw.client.0.smithi067.stdout: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0x6256851] 2019-12-10T06:10:47.473 INFO:tasks.rgw.client.0.smithi067.stdout: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x61d96e0] 2019-12-10T06:10:47.474 INFO:tasks.rgw.client.0.smithi067.stdout: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x61e6483] 2019-12-10T06:10:47.478 INFO:tasks.rgw.client.0.smithi067.stdout: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x61e6700] 2019-12-10T06:10:47.481 INFO:tasks.rgw.client.0.smithi067.stdout: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x61e6c5d] 2019-12-10T06:10:47.482 INFO:tasks.rgw.client.0.smithi067.stdout: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x61cf8e4] 2019-12-10T06:10:47.482 INFO:tasks.rgw.client.0.smithi067.stdout: 8: (AsyncConnection::process()+0x794) [0x619c534] 2019-12-10T06:10:47.483 INFO:tasks.rgw.client.0.smithi067.stdout: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0x61edc45] 2019-12-10T06:10:47.485 INFO:tasks.rgw.client.0.smithi067.stdout: 10: (()+0x571857) [0x61f5857] 2019-12-10T06:10:47.486 INFO:tasks.rgw.client.0.smithi067.stdout: 11: (()+0x844c6f) [0x64c8c6f] 2019-12-10T06:10:47.486 INFO:tasks.rgw.client.0.smithi067.stdout: 12: (()+0x7e65) [0x10088e65] 2019-12-10T06:10:47.486 INFO:tasks.rgw.client.0.smithi067.stdout: 13: (clone()+0x6d) [0x1128d88d] 2019-12-10T06:10:47.531 INFO:tasks.rgw.client.0.smithi067.stdout:2019-12-10T06:10:47.504+0000 18a15700 -1 *** Caught signal (Segmentation fault) ** 2019-12-10T06:10:47.531 INFO:tasks.rgw.client.0.smithi067.stdout: in thread 18a15700 thread_name:msgr-worker-1 2019-12-10T06:10:47.531 INFO:tasks.rgw.client.0.smithi067.stdout: 2019-12-10T06:10:47.532 INFO:tasks.rgw.client.0.smithi067.stdout: ceph version 15.0.0-7936-g827e03c (827e03c8a6b1e17d93dcaf79724437bac291c35b) octopus (dev) 2019-12-10T06:10:47.533 INFO:tasks.rgw.client.0.smithi067.stdout: 1: (()+0xf5f0) [0x100905f0] 2019-12-10T06:10:47.533 INFO:tasks.rgw.client.0.smithi067.stdout: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0x6256851] 2019-12-10T06:10:47.533 INFO:tasks.rgw.client.0.smithi067.stdout: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x61d96e0] 2019-12-10T06:10:47.533 INFO:tasks.rgw.client.0.smithi067.stdout: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x61e6483] 2019-12-10T06:10:47.535 INFO:tasks.rgw.client.0.smithi067.stdout: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x61e6700] 2019-12-10T06:10:47.535 INFO:tasks.rgw.client.0.smithi067.stdout: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x61e6c5d] 2019-12-10T06:10:47.536 INFO:tasks.rgw.client.0.smithi067.stdout: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x61cf8e4] 2019-12-10T06:10:47.537 INFO:tasks.rgw.client.0.smithi067.stdout: 8: (AsyncConnection::process()+0x794) [0x619c534] 2019-12-10T06:10:47.538 INFO:tasks.rgw.client.0.smithi067.stdout: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0x61edc45] 2019-12-10T06:10:47.538 INFO:tasks.rgw.client.0.smithi067.stdout: 10: (()+0x571857) [0x61f5857] 2019-12-10T06:10:47.538 INFO:tasks.rgw.client.0.smithi067.stdout: 11: (()+0x844c6f) [0x64c8c6f] 2019-12-10T06:10:47.538 INFO:tasks.rgw.client.0.smithi067.stdout: 12: (()+0x7e65) [0x10088e65] 2019-12-10T06:10:47.538 INFO:tasks.rgw.client.0.smithi067.stdout: 13: (clone()+0x6d) [0x1128d88d] 2019-12-10T06:10:47.540 INFO:tasks.rgw.client.0.smithi067.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-10T06:10:47.540 INFO:tasks.rgw.client.0.smithi067.stdout: 2019-12-10T06:10:48.884 INFO:tasks.rgw.client.0.smithi067.stdout: 0> 2019-12-10T06:10:47.504+0000 18a15700 -1 *** Caught signal (Segmentation fault) ** 2019-12-10T06:10:48.884 INFO:tasks.rgw.client.0.smithi067.stdout: in thread 18a15700 thread_name:msgr-worker-1 2019-12-10T06:10:48.885 INFO:tasks.rgw.client.0.smithi067.stdout: 2019-12-10T06:10:48.885 INFO:tasks.rgw.client.0.smithi067.stdout: ceph version 15.0.0-7936-g827e03c (827e03c8a6b1e17d93dcaf79724437bac291c35b) octopus (dev) 2019-12-10T06:10:48.885 INFO:tasks.rgw.client.0.smithi067.stdout: 1: (()+0xf5f0) [0x100905f0] 2019-12-10T06:10:48.885 INFO:tasks.rgw.client.0.smithi067.stdout: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0x6256851] 2019-12-10T06:10:48.885 INFO:tasks.rgw.client.0.smithi067.stdout: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x61d96e0] 2019-12-10T06:10:48.885 INFO:tasks.rgw.client.0.smithi067.stdout: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x61e6483] 2019-12-10T06:10:48.886 INFO:tasks.rgw.client.0.smithi067.stdout: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x61e6700] 2019-12-10T06:10:48.886 INFO:tasks.rgw.client.0.smithi067.stdout: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x61e6c5d] 2019-12-10T06:10:48.886 INFO:tasks.rgw.client.0.smithi067.stdout: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x61cf8e4] 2019-12-10T06:10:48.886 INFO:tasks.rgw.client.0.smithi067.stdout: 8: (AsyncConnection::process()+0x794) [0x619c534] 2019-12-10T06:10:48.886 INFO:tasks.rgw.client.0.smithi067.stdout: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0x61edc45] 2019-12-10T06:10:48.886 INFO:tasks.rgw.client.0.smithi067.stdout: 10: (()+0x571857) [0x61f5857] 2019-12-10T06:10:48.886 INFO:tasks.rgw.client.0.smithi067.stdout: 11: (()+0x844c6f) [0x64c8c6f] 2019-12-10T06:10:48.887 INFO:tasks.rgw.client.0.smithi067.stdout: 12: (()+0x7e65) [0x10088e65] 2019-12-10T06:10:48.887 INFO:tasks.rgw.client.0.smithi067.stdout: 13: (clone()+0x6d) [0x1128d88d] 2019-12-10T06:10:48.887 INFO:tasks.rgw.client.0.smithi067.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-10T06:10:48.887 INFO:tasks.rgw.client.0.smithi067.stdout: 2019-12-10T06:10:50.421 INFO:tasks.rgw.client.0.smithi067.stdout: 0> 2019-12-10T06:10:47.504+0000 18a15700 -1 *** Caught signal (Segmentation fault) ** 2019-12-10T06:10:50.421 INFO:tasks.rgw.client.0.smithi067.stdout: in thread 18a15700 thread_name:msgr-worker-1 2019-12-10T06:10:50.421 INFO:tasks.rgw.client.0.smithi067.stdout: 2019-12-10T06:10:50.421 INFO:tasks.rgw.client.0.smithi067.stdout: ceph version 15.0.0-7936-g827e03c (827e03c8a6b1e17d93dcaf79724437bac291c35b) octopus (dev) 2019-12-10T06:10:50.422 INFO:tasks.rgw.client.0.smithi067.stdout: 1: (()+0xf5f0) [0x100905f0] 2019-12-10T06:10:50.422 INFO:tasks.rgw.client.0.smithi067.stdout: 2: (MonClient::handle_auth_done(Connection*, AuthConnectionMeta*, unsigned long, unsigned int, ceph::buffer::v14_2_0::list const&, CryptoKey*, std::string*)+0x71) [0x6256851] 2019-12-10T06:10:50.422 INFO:tasks.rgw.client.0.smithi067.stdout: 3: (ProtocolV2::handle_auth_done(ceph::buffer::v14_2_0::list&)+0x2e0) [0x61d96e0] 2019-12-10T06:10:50.422 INFO:tasks.rgw.client.0.smithi067.stdout: 4: (ProtocolV2::handle_frame_payload()+0x1d3) [0x61e6483] 2019-12-10T06:10:50.422 INFO:tasks.rgw.client.0.smithi067.stdout: 5: (ProtocolV2::handle_read_frame_dispatch()+0x150) [0x61e6700] 2019-12-10T06:10:50.422 INFO:tasks.rgw.client.0.smithi067.stdout: 6: (ProtocolV2::handle_read_frame_epilogue_main(std::unique_ptr&&, int)+0x44d) [0x61e6c5d] 2019-12-10T06:10:50.422 INFO:tasks.rgw.client.0.smithi067.stdout: 7: (ProtocolV2::run_continuation(Ct&)+0x34) [0x61cf8e4] 2019-12-10T06:10:50.423 INFO:tasks.rgw.client.0.smithi067.stdout: 8: (AsyncConnection::process()+0x794) [0x619c534] 2019-12-10T06:10:50.423 INFO:tasks.rgw.client.0.smithi067.stdout: 9: (EventCenter::process_events(unsigned int, std::chrono::duration >*)+0xa25) [0x61edc45] 2019-12-10T06:10:50.424 INFO:tasks.rgw.client.0.smithi067.stdout: 10: (()+0x571857) [0x61f5857] 2019-12-10T06:10:50.424 INFO:tasks.rgw.client.0.smithi067.stdout: 11: (()+0x844c6f) [0x64c8c6f] 2019-12-10T06:10:50.424 INFO:tasks.rgw.client.0.smithi067.stdout: 12: (()+0x7e65) [0x10088e65] 2019-12-10T06:10:50.425 INFO:tasks.rgw.client.0.smithi067.stdout: 13: (clone()+0x6d) [0x1128d88d] 2019-12-10T06:10:50.425 INFO:tasks.rgw.client.0.smithi067.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2019-12-10T06:10:50.436 INFO:tasks.rgw.client.0.smithi067.stdout: 2019-12-10T06:11:06.551 INFO:tasks.rgw.client.0.smithi067.stderr:daemon-helper: command crashed with signal 11 2019-12-10T06:11:08.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:11:08.634+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:11:11.916 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:37:32.978 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:11.921 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:37:32.983 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:11.930 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:37:32.993 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:11.934 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:37:32.997 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:11.947 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:37:33.108 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:11.965 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:37:33.126 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:12.128 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:37:33.190 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:12.151 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:37:33.213 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:12.231 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:37:33.391 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:30.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:11:30.634+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:11:40.636 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:38:01.698 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:40.655 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:38:01.717 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:40.718 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:38:01.878 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:40.720 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:38:01.782 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:40.721 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:38:01.882 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:40.726 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:38:01.789 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:40.734 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:38:01.796 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:40.761 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:38:01.824 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:40.788 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:38:01.949 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:11:52.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:11:52.634+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:12:14.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:12:14.634+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:12:15.840 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:12:15.838+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe4700118d0 msgr2=0x7fe47003efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T06:12:36.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:12:36.634+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:12:58.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:12:58.635+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:13:20.636 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:13:20.635+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:13:42.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:13:42.634+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:14:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:14:04.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:14:26.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:14:26.635+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:14:48.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:14:48.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:15:10.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:15:10.635+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:15:32.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:15:32.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:15:54.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:15:54.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:16:16.637 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:16:16.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:16:20.357 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:42:41.419 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:16:20.366 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:42:41.429 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:16:20.377 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:42:41.439 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:16:20.378 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:42:41.539 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:16:20.382 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:42:41.444 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:16:20.384 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:42:41.545 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:16:20.575 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:42:41.638 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:16:20.578 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:42:41.739 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:16:20.594 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:42:41.655 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:16:38.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:16:38.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:17:00.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:17:00.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:17:22.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:17:22.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:17:44.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:17:44.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:17:47.874 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:17:47.873+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x7fe47000de40 msgr2=0x7fe4700348b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T06:18:06.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:18:06.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:18:28.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:18:28.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:18:50.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:18:50.636+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:19:12.638 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:19:12.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:19:34.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:19:34.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:19:56.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:19:56.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:20:02.848 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:46:23.910 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:20:02.863 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:46:23.926 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:20:02.925 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:46:23.987 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:20:02.929 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:46:23.992 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:20:02.940 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:46:24.098 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:20:02.941 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:46:24.101 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:20:03.085 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:46:24.147 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:20:03.092 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:46:24.154 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:20:03.094 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:46:24.255 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:20:18.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:20:18.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:20:40.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:20:40.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:21:02.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:21:02.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:21:24.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:21:24.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:21:32.701 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:47:53.763 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:21:32.712 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:47:53.775 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:21:32.724 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:47:53.885 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:21:32.726 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:47:53.789 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:21:32.734 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:47:53.797 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:21:32.735 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:47:53.896 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:21:32.853 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:47:54.014 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:21:32.856 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:47:53.918 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:21:32.878 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:47:53.940 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:21:46.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:21:46.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:22:08.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:22:08.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:22:30.639 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:22:30.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:22:52.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:22:52.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:23:14.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:23:14.637+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:23:36.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:23:36.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:23:58.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:23:58.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:24:20.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:24:20.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:24:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:24:42.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:25:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:25:04.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:25:26.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:25:26.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:25:48.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:25:48.638+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:26:10.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:26:10.639+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:26:32.640 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:26:32.639+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:26:39.842 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:53:00.904 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:39.884 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:53:00.946 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:39.962 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:53:01.025 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:39.964 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:53:01.026 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:39.969 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:53:01.129 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:39.982 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:53:01.143 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:40.136 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:53:01.199 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:40.182 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:53:01.343 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:40.215 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:53:01.278 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:45.170 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:53:06.233 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:45.182 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:53:06.244 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:45.195 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:53:06.258 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:45.197 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:53:06.358 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:45.202 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:53:06.265 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:45.213 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:53:06.373 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:45.216 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:53:06.279 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:45.257 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:53:06.418 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:45.260 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:53:06.323 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:26:54.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:26:54.639+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:27:16.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:27:16.640+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:27:32.980 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:27:32.979+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe470009100 msgr2=0x7fe470049b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T06:27:38.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:27:38.639+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:28:00.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:28:00.640+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:28:07.941 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:28:07.940+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe470004470 msgr2=0x7fe470006890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T06:28:22.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:28:22.639+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:28:25.635 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:54:46.698 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:28:25.639 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:54:46.702 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:28:25.701 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:54:46.763 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:28:25.710 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:54:46.773 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:28:25.713 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:54:46.775 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:28:25.722 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:54:46.883 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:28:25.724 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:54:46.885 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:28:25.784 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:54:46.846 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:28:25.796 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:54:46.957 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:28:44.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:28:44.640+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:29:06.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:29:06.640+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:29:28.641 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:29:28.640+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:29:50.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:29:50.641+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:30:12.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:30:12.640+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:30:34.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:30:34.641+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:30:56.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:30:56.641+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:31:18.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:31:18.641+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:31:40.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:31:40.641+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:31:57.523 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:58:18.586 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:31:57.530 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:58:18.593 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:31:57.538 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:58:18.699 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:31:57.538 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:58:18.601 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:31:57.541 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:58:18.603 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:31:57.544 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:58:18.705 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:31:57.708 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:09:58:18.869 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:31:57.725 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:09:58:18.788 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:31:57.762 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:09:58:18.825 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:32:02.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:32:02.641+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:32:24.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:32:24.641+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:32:46.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:32:46.642+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:32:47.991 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:32:47.990+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe47000de40 msgr2=0x7fe4700348b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T06:33:08.642 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:33:08.641+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:33:30.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:33:30.642+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:33:52.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:33:52.641+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:34:14.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:34:14.642+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:34:36.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:34:36.642+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:34:58.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:34:58.642+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:35:20.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:35:20.642+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:35:42.643 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:35:42.643+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:36:04.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:36:04.642+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:36:26.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:36:26.643+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:36:48.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:36:48.643+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:36:52.793 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:03:13.856 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:36:52.797 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:03:13.860 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:36:52.859 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:03:14.020 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:36:52.861 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:03:14.022 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:36:52.999 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:03:14.159 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:36:53.025 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:03:14.083 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:36:53.055 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:03:14.118 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:36:53.088 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:03:14.151 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:36:53.230 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:03:14.293 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:07.771 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:03:28.834 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:07.777 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:03:28.840 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:07.789 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:03:28.852 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:07.795 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:03:28.858 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:07.803 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:03:28.964 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:07.821 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:03:28.981 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:07.832 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:03:28.895 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:07.854 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:03:28.917 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:07.919 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:03:29.080 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:37:10.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:37:10.643+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:37:32.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:37:32.643+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:37:54.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:37:54.643+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:38:16.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:38:16.643+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:38:33.079 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:38:33.078+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6812/9679,v1:172.21.15.81:6814/9679] conn(0x7fe470034df0 msgr2=0x7fe47004d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T06:38:38.644 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:38:38.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:39:00.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:39:00.643+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:39:22.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:39:22.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:39:44.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:39:44.643+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:40:06.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:40:06.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:40:28.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:40:28.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:40:50.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:40:50.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:41:12.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:41:12.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:41:34.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:41:34.645+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:41:56.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:41:56.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:42:18.645 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:42:18.645+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:42:19.655 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:08:40.718 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:42:19.662 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:08:40.725 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:42:19.671 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:08:40.832 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:42:19.674 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:08:40.734 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:42:19.675 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:08:40.836 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:42:19.676 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:08:40.739 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:42:19.882 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:08:41.041 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:42:19.908 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:08:40.971 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:42:19.932 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:08:40.995 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:42:40.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:42:40.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:43:02.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:43:02.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:43:24.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:43:24.644+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:43:46.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:43:46.645+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:44:08.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:44:08.645+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:44:30.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:44:30.645+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:44:52.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:44:52.645+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:45:14.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:45:14.646+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:45:15.517 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:11:36.580 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:45:15.519 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:11:36.583 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:45:15.599 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:11:36.662 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:45:15.603 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:11:36.666 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:45:15.605 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:11:36.766 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:45:15.607 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:11:36.768 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:45:15.631 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:11:36.694 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:45:15.709 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:11:36.772 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:45:15.709 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:11:36.870 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:45:36.646 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:45:36.645+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:45:58.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:45:58.645+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:46:20.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:46:20.646+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:46:42.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:46:42.645+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:47:04.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:47:04.647+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:47:26.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:47:26.646+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:47:31.093 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:13:52.156 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:47:31.102 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:13:52.166 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:47:31.117 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:13:52.180 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:47:31.119 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:13:52.280 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:47:31.128 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:13:52.191 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:47:31.130 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:13:52.291 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:47:31.231 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:13:52.294 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:47:31.242 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:13:52.403 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:47:31.244 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:13:52.307 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:47:48.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:47:48.647+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:48:10.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:48:10.646+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:48:32.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:48:32.647+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:48:54.647 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:48:54.647+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:49:16.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:49:16.647+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:49:38.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:49:38.647+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:50:00.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:50:00.648+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:50:22.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:50:22.647+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:50:44.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:50:44.648+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:51:06.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:51:06.647+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:51:28.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:51:28.648+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:51:50.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:51:50.647+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:52:12.648 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:52:12.648+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:52:34.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:52:34.648+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:52:38.250 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:52:38.249+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe4700069c0 msgr2=0x7fe4700087b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T06:52:44.472 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:19:05.535 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:52:44.484 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:19:05.645 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:52:44.486 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:19:05.550 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:52:44.490 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:19:05.651 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:52:44.511 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:19:05.567 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:52:44.523 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:19:05.586 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:52:44.736 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:19:05.897 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:52:44.743 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:19:05.807 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:52:44.756 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:19:05.819 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:52:56.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:52:56.649+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:53:18.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:53:18.648+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:53:37.744 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:19:58.905 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:53:37.753 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:19:58.817 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:53:37.779 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:19:58.842 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:53:37.800 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:19:58.961 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:53:37.816 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:19:58.879 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:53:37.818 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:19:58.881 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:53:37.847 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:19:58.907 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:53:37.854 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:19:59.015 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:53:37.874 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:19:58.938 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:53:40.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:53:40.649+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:54:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:54:02.648+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:54:24.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:54:24.649+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:54:46.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:54:46.649+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:55:08.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:55:08.649+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:55:30.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:55:30.649+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:55:52.649 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:55:52.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:56:14.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:56:14.649+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:56:36.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:56:36.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:56:58.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:56:58.649+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:57:20.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:57:20.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:57:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:57:42.649+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:57:56.034 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:24:17.092 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:57:56.039 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:24:17.103 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:57:56.052 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:24:17.115 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:57:56.055 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:24:17.216 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:57:56.057 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:24:17.121 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:57:56.066 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:24:17.227 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:57:56.226 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:24:17.290 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T06:57:56.239 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:24:17.302 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T06:57:56.288 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:24:17.450 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T06:58:04.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:58:04.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:58:26.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:58:26.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:58:48.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:58:48.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:59:10.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:59:10.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:59:32.650 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:59:32.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T06:59:53.300 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:59:53.296+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe47000de40 msgr2=0x7fe4700348b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T06:59:54.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T06:59:54.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:00:16.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:00:16.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:00:35.889 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:00:35.888+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:3300/0,v1:172.21.15.81:6789/0] conn(0x7fe4700249c0 msgr2=0x7fe470009cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T07:00:38.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:00:38.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:01:00.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:01:00.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:01:22.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:01:22.650+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:01:44.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:01:44.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:01:59.885 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:28:20.949 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:01:59.942 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:28:21.006 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:01:59.945 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:28:21.009 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:01:59.957 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:28:21.020 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:02:00.039 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:28:21.200 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:02:00.051 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:28:21.212 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:02:00.075 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:28:21.139 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:02:00.091 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:28:21.155 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:02:00.187 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:28:21.349 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:02:06.651 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:02:06.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:02:28.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:02:28.652+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:02:50.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:02:50.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:03:06.084 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:29:27.148 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:03:06.098 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:29:27.161 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:03:06.110 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:29:27.173 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:03:06.120 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:29:27.281 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:03:06.133 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:29:27.197 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:03:06.157 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:29:27.318 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:03:06.224 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:29:27.287 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:03:06.249 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:29:27.312 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:03:06.297 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:29:27.458 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:03:12.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:03:12.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:03:34.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:03:34.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:03:56.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:03:56.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:04:18.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:04:18.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:04:40.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:04:40.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:05:02.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:05:02.651+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:05:24.652 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:05:24.652+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:05:46.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:05:46.652+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:06:08.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:06:08.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:06:30.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:06:30.652+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:06:43.377 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:06:43.376+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x7fe47004a0b0 msgr2=0x7fe470010440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T07:06:52.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:06:52.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:07:06.570 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:33:27.632 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:07:06.610 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:33:27.673 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:07:06.711 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:33:27.775 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:07:06.715 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:33:27.779 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:07:06.772 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:33:27.933 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:07:06.775 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:33:27.937 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:07:06.827 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:33:27.891 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:07:06.895 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:33:27.959 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:07:07.001 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:33:28.162 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:07:14.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:07:14.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:07:36.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:07:36.652+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:07:58.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:07:58.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:08:17.040 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:34:38.104 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:08:17.056 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:34:38.120 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:08:17.077 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:34:38.141 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:08:17.078 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:34:38.240 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:08:17.084 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:34:38.246 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:08:17.085 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:34:38.149 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:08:17.143 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:34:38.206 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:08:17.180 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:34:38.244 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:08:17.195 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:34:38.357 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:08:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:08:20.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:08:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:08:42.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:09:04.653 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:09:04.654+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:09:26.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:09:26.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:09:48.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:09:48.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:10:10.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:10:10.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:10:22.636 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:36:43.699 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:10:22.643 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:36:43.707 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:10:22.684 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:36:43.748 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:10:22.716 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:36:43.780 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:10:22.717 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:36:43.878 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:10:22.721 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:36:43.883 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:10:22.749 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:36:43.808 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:10:22.802 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:36:43.963 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:10:22.820 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:36:43.884 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:10:32.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:10:32.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:10:54.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:10:54.653+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:11:16.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:11:16.654+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:11:38.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:11:38.654+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:12:00.654 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:12:00.655+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:12:22.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:12:22.654+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:12:44.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:12:44.654+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:13:06.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:13:06.654+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:13:28.421 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:39:49.485 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:13:28.424 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:39:49.488 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:13:28.445 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:39:49.607 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:13:28.453 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:39:49.614 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:13:28.493 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:39:49.557 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:13:28.505 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:39:49.569 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:13:28.611 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:39:49.675 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:13:28.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:13:28.654+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:13:28.659 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:39:49.820 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:13:28.662 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:39:49.725 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:13:50.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:13:50.655+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:14:05.902 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:14:05.901+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe47000de40 msgr2=0x7fe4700348b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T07:14:12.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:14:12.654+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:14:34.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:14:34.656+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:14:56.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:14:56.655+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:15:18.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:15:18.656+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:15:40.655 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:15:40.655+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:16:02.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:16:02.656+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:16:24.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:16:24.656+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:16:46.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:16:46.657+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:17:08.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:17:08.656+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:17:30.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:17:30.657+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:17:52.656 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:17:52.657+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:18:14.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:18:14.657+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:18:36.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:18:36.657+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:18:40.709 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:45:01.773 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:40.719 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:45:01.783 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:40.731 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:45:01.795 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:40.742 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:45:01.807 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:40.747 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:45:01.909 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:40.757 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:45:01.918 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:40.989 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:45:02.053 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:40.990 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:45:02.054 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:41.065 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:45:02.226 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:44.655 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:45:05.719 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:44.663 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:45:05.727 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:44.686 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:45:05.750 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:44.687 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:45:05.751 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:44.743 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:45:05.904 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:44.749 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:45:05.910 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:44.751 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:45:05.814 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:44.769 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:45:05.833 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:44.770 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:45:05.931 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:18:58.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:18:58.657+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:19:20.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:19:20.657+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:19:42.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:19:42.658+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:20:04.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:20:04.657+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:20:26.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:20:26.658+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:20:48.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:20:48.658+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:21:10.657 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:21:10.658+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:21:32.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:21:32.657+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:21:54.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:21:54.658+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:22:16.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:22:16.658+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:22:38.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:22:38.659+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:23:00.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:23:00.658+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:23:22.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:23:22.659+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:23:44.658 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:23:44.658+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:23:52.125 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:50:13.189 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:23:52.140 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:50:13.204 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:23:52.146 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:50:13.210 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:23:52.151 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:50:13.215 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:23:52.153 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:50:13.314 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:23:52.159 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:50:13.320 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:23:52.381 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:50:13.445 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:23:52.419 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:50:13.483 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:23:52.471 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:50:13.631 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:24:06.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:24:06.659+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:24:28.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:24:28.659+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:24:50.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:24:50.659+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:25:12.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:25:12.659+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:25:34.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:25:34.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:25:56.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:25:56.659+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:26:18.659 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:26:18.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:26:40.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:26:40.659+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:27:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:27:02.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:27:06.865 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:53:27.926 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:27:06.869 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:53:27.933 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:27:06.907 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:53:27.971 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:27:06.911 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:53:27.975 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:27:06.935 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:53:28.095 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:27:06.937 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:53:28.098 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:27:06.980 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:53:28.044 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:27:07.036 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:53:28.100 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:27:07.066 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:53:28.227 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:27:24.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:27:24.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:27:46.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:27:46.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:28:08.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:28:08.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:28:30.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:28:30.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:28:38.632 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:28:38.632+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x7fe470008cf0 msgr2=0x7fe470049af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T07:28:52.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:28:52.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:29:02.478 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:55:23.542 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:29:02.489 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:55:23.553 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:29:02.502 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:55:23.566 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:29:02.505 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:55:23.666 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:29:02.512 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:55:23.673 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:29:02.516 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:55:23.580 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:29:02.596 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:10:55:23.660 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:29:02.610 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:10:55:23.674 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:29:02.647 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:10:55:23.807 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:29:14.660 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:29:14.661+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:29:36.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:29:36.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:29:58.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:29:58.661+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:30:13.607 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:30:13.607+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x7fe47000c060 msgr2=0x7fe47000c470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T07:30:20.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:30:20.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:30:42.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:30:42.661+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:31:04.661 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:31:04.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:31:26.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:31:26.661+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:31:48.625 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:31:48.624+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe470009100 msgr2=0x7fe470006480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T07:31:48.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:31:48.660+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:31:58.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:31:58.666+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x7fe470008cf0 msgr2=0x7fe470049af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T07:32:10.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:32:10.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:32:32.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:32:32.661+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:32:54.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:32:54.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:33:16.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:33:16.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:33:38.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:33:38.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:34:00.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:34:00.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:34:14.869 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:00:35.932 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:34:14.877 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:00:35.941 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:34:14.895 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:00:35.959 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:34:14.898 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:00:36.058 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:34:14.906 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:00:35.970 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:34:14.916 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:00:36.077 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:34:15.090 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:00:36.154 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:34:15.185 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:00:36.345 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:34:15.192 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:00:36.256 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:34:22.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:34:22.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:34:44.662 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:34:44.661+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:35:06.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:35:06.663+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:35:28.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:35:28.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:35:29.090 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:01:50.154 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:35:29.093 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:01:50.157 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:35:29.155 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:01:50.316 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:35:29.158 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:01:50.319 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:35:29.170 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:01:50.233 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:35:29.174 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:01:50.238 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:35:29.207 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:01:50.270 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:35:29.253 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:01:50.317 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:35:29.255 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:01:50.416 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:35:50.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:35:50.663+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:36:12.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:36:12.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:36:34.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:36:34.663+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:36:56.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:36:56.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:37:18.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:37:18.663+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:37:40.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:37:40.662+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:38:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:38:02.664+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:38:13.694 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:38:13.694+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe47000c060 msgr2=0x7fe47000c470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T07:38:24.663 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:38:24.663+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:38:46.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:38:46.664+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:39:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:39:08.663+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:39:26.274 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:05:47.338 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:39:26.293 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:05:47.354 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:39:26.303 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:05:47.458 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:39:26.303 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:05:47.361 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:39:26.312 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:05:47.468 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:39:26.318 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:05:47.379 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:39:26.490 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:05:47.554 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:39:26.521 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:05:47.584 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:39:26.545 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:05:47.705 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:39:30.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:39:30.664+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:39:52.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:39:52.663+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:40:14.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:40:14.664+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:40:36.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:40:36.664+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:40:58.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:40:58.664+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:41:20.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:41:20.664+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:41:42.664 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:41:42.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:42:04.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:42:04.664+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:42:26.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:42:26.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:42:48.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:42:48.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:43:10.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:43:10.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:43:32.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:43:32.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:43:51.393 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:10:12.457 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:43:51.407 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:10:12.471 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:43:51.476 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:10:12.540 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:43:51.492 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:10:12.651 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:43:51.494 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:10:12.654 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:43:51.497 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:10:12.561 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:43:51.578 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:10:12.642 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:43:51.636 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:10:12.699 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:43:51.646 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:10:12.807 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:43:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:43:54.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:44:16.665 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:44:16.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:44:36.628 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:10:57.691 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:44:36.637 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:10:57.701 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:44:36.654 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:10:57.717 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:44:36.664 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:10:57.824 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:44:36.667 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:10:57.731 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:44:36.681 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:10:57.841 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:44:36.717 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:10:57.781 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:44:36.743 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:10:57.805 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:44:36.796 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:10:57.956 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:44:38.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:44:38.666+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:44:38.806 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:44:38.806+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe470008cf0 msgr2=0x7fe470049af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T07:45:00.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:45:00.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:45:22.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:45:22.666+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:45:44.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:45:44.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:46:06.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:46:06.666+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:46:28.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:46:28.665+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:46:50.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:46:50.666+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:47:12.666 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:47:12.666+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:47:34.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:47:34.667+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:47:56.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:47:56.666+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:48:07.780 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:14:28.844 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:48:07.817 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:14:28.881 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:48:07.861 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:14:28.925 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:48:07.863 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:14:28.927 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:48:07.908 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:14:29.068 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:48:07.911 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:14:29.072 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:48:07.992 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:14:29.056 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:48:08.019 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:14:29.083 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:48:08.100 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:14:29.261 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:48:18.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:48:18.667+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:48:40.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:48:40.667+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:49:02.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:49:02.667+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:49:24.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:49:24.667+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:49:46.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:49:46.668+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:49:48.255 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:16:09.319 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:49:48.296 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:16:09.443 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:49:48.296 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:16:09.339 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:49:48.305 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:16:09.365 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:49:48.305 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:16:09.462 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:49:48.324 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:16:09.380 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:49:48.423 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:16:09.583 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:49:48.424 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:16:09.487 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:49:48.432 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:16:09.496 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:50:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:50:08.667+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:50:30.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:50:30.668+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:50:52.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:50:52.667+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:51:14.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:51:14.668+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:51:36.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:51:36.668+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:51:58.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:51:58.668+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:52:13.905 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:18:34.968 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:52:13.910 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:18:34.974 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:52:13.955 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:18:35.019 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:52:13.960 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:18:35.024 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:52:13.986 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:18:35.146 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:52:13.995 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:18:35.155 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:52:14.002 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:18:35.066 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:52:14.047 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:18:35.110 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:52:14.162 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:18:35.322 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:52:20.668 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:52:20.668+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:52:42.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:52:42.669+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:53:04.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:53:04.668+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:53:26.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:53:26.669+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:53:48.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:53:48.668+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:54:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:54:10.669+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:54:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:54:13.902+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x7fe470008cf0 msgr2=0x7fe470049af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T07:54:32.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:54:32.668+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:54:54.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:54:54.669+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:54:59.908 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:21:20.972 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:54:59.916 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:21:20.980 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:54:59.929 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:21:20.993 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:54:59.932 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:21:21.093 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:54:59.950 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:21:21.014 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:54:59.962 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:21:21.123 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:55:00.055 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:21:21.119 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T07:55:00.121 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:21:21.184 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T07:55:00.151 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:21:21.312 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T07:55:16.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:55:16.669+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:55:38.669 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:55:38.670+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:56:00.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:56:00.669+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:56:22.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:56:22.670+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:56:44.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:56:44.669+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:57:06.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:57:06.670+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:57:28.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:57:28.670+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:57:50.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:57:50.670+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:58:12.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:58:12.669+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:58:34.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:58:34.671+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:58:56.670 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:58:56.670+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:59:18.671 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:59:18.671+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T07:59:40.671 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T07:59:40.670+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:00:02.671 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:00:02.671+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:00:11.998 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:26:33.061 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:12.008 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:26:33.071 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:12.028 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:26:33.188 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:12.033 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:26:33.096 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:12.038 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:26:33.199 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:12.043 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:26:33.106 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:12.339 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:26:33.398 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:12.339 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:26:33.491 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:12.358 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:26:33.422 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:24.671 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:00:24.671+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:00:36.040 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:26:57.104 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:36.053 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:26:57.113 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:36.096 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:26:57.159 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:36.100 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:26:57.164 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:36.110 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:26:57.174 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:36.124 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:26:57.284 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:36.127 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:26:57.288 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:36.131 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:26:57.195 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:36.179 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:26:57.340 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:00:46.671 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:00:46.671+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:01:08.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:01:08.671+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:01:30.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:01:30.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:01:52.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:01:52.671+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:01:58.969 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:01:58.968+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe470002b30 msgr2=0x7fe470033920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T08:02:08.991 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:02:08.990+0000 7fe487fff700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6816/9681,v1:172.21.15.81:6818/9681] conn(0x7fe470034df0 msgr2=0x7fe47004d330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2019-12-10T08:02:14.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:02:14.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:02:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:02:36.671+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:02:58.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:02:58.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:03:20.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:03:20.671+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:03:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:03:42.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:04:04.672 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:04:04.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:04:26.676 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:04:26.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:04:48.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:04:48.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:05:10.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:05:10.673+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:05:24.331 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:31:45.394 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:05:24.342 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:31:45.406 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:05:24.358 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:31:45.518 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:05:24.359 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:31:45.423 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:05:24.376 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:31:45.537 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:05:24.377 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:31:45.441 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:05:24.559 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:31:45.623 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:05:24.615 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:31:45.678 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:05:24.640 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:31:45.800 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:05:32.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:05:32.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:05:54.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:05:54.673+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:06:16.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:06:16.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:06:18.997 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:06:18.996+0000 7fe48fe99700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.81:6808/9678,v1:172.21.15.81:6809/9678] conn(0x7fe47004a030 msgr2=0x7fe470010440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T08:06:38.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:06:38.673+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:07:00.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:07:00.673+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:07:22.673 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:07:22.673+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:07:44.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:07:44.672+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:08:06.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:08:06.674+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:08:28.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:08:28.673+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:08:50.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:08:50.674+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:08:58.149 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:35:19.212 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:08:58.156 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:35:19.220 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:08:58.248 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:35:19.408 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:08:58.252 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:35:19.316 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:08:58.253 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:35:19.414 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:08:58.256 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:35:19.319 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:08:58.364 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:35:19.524 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:08:58.406 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:35:19.470 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:08:58.436 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:35:19.500 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:09:12.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:09:12.674+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:09:34.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:09:34.674+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:09:56.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:09:56.674+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:10:18.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:10:18.674+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:10:35.856 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:36:56.920 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:10:35.861 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:36:56.925 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:10:35.893 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:36:57.048 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:10:35.893 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:36:56.951 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:10:35.901 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:36:57.057 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:10:35.901 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:36:56.961 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:10:35.958 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:11:36:57.022 9310== Warning: unimplemented fcntl command: 1036 2019-12-10T08:10:36.011 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:11:36:57.075 9309== Warning: unimplemented fcntl command: 1036 2019-12-10T08:10:36.036 INFO:tasks.ceph.mon.b.smithi081.stderr:==00:11:36:57.197 8820== Warning: unimplemented fcntl command: 1036 2019-12-10T08:10:40.674 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:10:40.674+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:11:02.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:11:02.675+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:11:24.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:11:24.674+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:11:46.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:11:46.675+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:12:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:12:08.675+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:12:30.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:12:30.675+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:12:39.064 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:12:39.064+0000 7fe48f698700 0 -- 172.21.15.67:0/4236188761 >> [v2:172.21.15.67:6805/10703,v1:172.21.15.67:6806/10703] conn(0x7fe470009100 msgr2=0x7fe470006480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2019-12-10T08:12:52.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:12:52.675+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:13:14.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:13:14.675+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:13:36.675 INFO:teuthology.orchestra.run.smithi067.stderr:2019-12-10T08:13:36.675+0000 7fe45ffdf700 2 RGWDataChangesLog::ChangesRenewThread: start 2019-12-10T08:13:46.413 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-12-10T08:13:46.455 DEBUG:teuthology.task.console_log:Killing console logger for smithi067 2019-12-10T08:13:46.570 DEBUG:teuthology.task.console_log:Killing console logger for smithi081 2019-12-10T08:13:46.581 DEBUG:teuthology.task.console_log:Killing console logger for smithi067 2019-12-10T08:13:46.586 DEBUG:teuthology.task.console_log:Killing console logger for smithi081 2019-12-10T08:13:46.591 DEBUG:teuthology.exit:Finished running handlers